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PREFACE 



When Alverno revised its liberal arts curriculum in 1973 to focus on abilities, the college was 
immediately asked a series of questions from all quarters: Does the curriculum work? Does the 
curriculum contribute to student outcomes-and to graduates in the world bf work? What have 
you learned about student learning and development that can inform issues in higher education 
and generalize to other settings? In 1976, the second author of this report invited the first 
author to initiate, create and direct a college-wide effort to respond to these questions. We began 
longitudinal studies that fall. A year later, we garnered three years of financial support from the 
National Institute of Education to carry out a validation model and created an Office of Research 
and Evaluation whrch is now funded by the college. 

During our dissemination of preliminary findings, we found that the questions surround nj 
curriculum innovation and institutional reform in the seventies are even more alive in the eightie j, 
as the nation expresses its concern for qualii/ and excellence in education. Questioners in the 
seventies, who wondered aloud whether changes in liberal arts curricula would positively affect 
learning outcomes, have become more focused, and more demanding of explicit and practical 
answers. They are less interested in maps for total institutional reform. Rather, they are asking 
for descriptions of student development and abilities and of teaching and assessment stiategies 
that ensure high performance learning in diverse settings, for diverse groups. 

Thus, we highlight those findings that speak to these questions, and that guide our current 
studies. In fact, this second edition of the overview and summary expands on these findings, 
following last year's presentations at specially convened seminars sponsored by the American 
Association for Higher Education, the National Institute of F^ucation and the Carnegie 
Corporation of New York. 

During these and other presentations, many of our colleagues expressed interest in the broader 
issue of validation. Concerned about validating their own programs, they wished to know how 
we conceptualized the validation of a liberal arts, outcome-centered curriculum. .We Lave 
responded to this interest by describing the more detailed faculty questions that stimulated 
the initial study of college outcomes .\t Alverno, in the context of the validation model that has 
guided our efforts. Thus, this overview and summary first describes the rationale for validating 
outcome-centered higher education curricula and our validation model drawn from faculty 
questions. We then describe the research objectives based on these questions and our approacljes 
to instrumentation and methodology. 

In the last tvyo sections of the report, "What are the Outcomes of an Alverno College Experience," 
and "How do Alverno College Outcomes Relate to the World of Work," we summarize the last 
eight years of results from our studies of learning outcomes. We then relate the findings to our 
overall purposes and discuss implications for higher education. These sections, including "New 
Directions " ana "Summary," are printed on gray paper (pages 95 to 165), and are designed for 
the reader who wants an immediate overview. This synthesis is drawn fiom the ten research 
reports listed on the inside cover that form our final report to the National Institute of Education, 
as well as our other more recent reports. We also include abstracts of the research repoits, identify 
our dissemination strategies and list the range of colleges and universities, corporations, agencies 
and schools whose questions and insights have contributed to our efforts during these past eight 
years. 



This research represents the collaborative work of the Alverno faculty, Office of Research and 
Evaluation staff, Alverno students and alumnae, and Milwaukee organizations and professionals. 
Our acknowledgements to them and to other colleagues follow the preface. 

This work is dedicated to our students, whose belief in our ability to improve education gives us. 
the faith and courage to continue learning, and to research the penetrating questions that chaiiengt; 
higher education today. 

Marcia Mentkowski 
Austin Doherty 



Milwaukee, Wisconsin 
May 1984 
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CAREERING AFTER COLLEGE 
ESTABLISHING THE VALIDITY OF ABILITIES LEARNED IN COLLEGE 
FOR LATER CAREERING AND PROFESSIONAL PERFORMANCE 

Marcia Mentkowski Austin Doherty 

ALVERNO COLLEGE 

ABSTRACT 



Wliat differences does college make? Can it really promote 
the kind of broad personal and intellectual development that 
lasts a lifetime? Can it enhance a person's abilities and 
improv'e his or her chances at having an effective career.' Can it 
benefit the "new" student body — adults, women, minorities — as 
well as traditional college ^^udents? Do the outcomes of college 
show up on the job? 

That students change in college is taken for granted by most 
college educators and has been demonstrated by several 
researchers of college outcomes (Astin, 1977; Feldraan & Newcomb, 
1969; Heath, 1977; Pace, 1979; Perry, 1970; Vaillant, 1977; 
Winter, McClelland & Stewart, 1981). That students change in 
college as the result' of performance in a particular curriculum 
is more difficult to show. How students change, and who changes 
and why — and with respect to wha" btoad,. complex abilities, 
learning styles and cognitive-developmental patterns (Chickering 
fi Associates, 1981) — is even a^re illusive . Demonstrating that 
these changes persist beyond c liege to effective performance in 
work and persanal roles is perhaps most challenging of all. 
Showing that abilities selected by college faculty and 
demonstrated by their students are used by outstanding 
professionals in the world of work, is clearly a new issue for 
college educators. 

^et these are precisely the issues raised by one liberal arts 
college faculty who broke with tradition and implemented an 



^The faculty we have been working with are our colleagues at 
I Alverno College, a Midwestern liberal arts college for women 

I with about 1400 degree students in both weekday and weekend time 

frames. Alverno, which has focused for a century on preparing 
m women for professional careers, f&rmally adopted an out v-ne 

I centered approach to ics curriculum in 1973, accrediting students 

' for progressive demonstration of certain broad abilities in all 

subject areas, 
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outcome-centered liberal arts curriculum in 1973. The taculty 
identified broad outcomes promised by many colleges. But they 
defined these complex abilities through a set of pedagogical 
levels that allowed for their successive achievement, and created 
measures that assessed not only knowledge, but the student's 
performance. When the first students were about to graduate from 
the new curriculum, the faculty engaged in a multifaceted attempt 

7 to focus on the external validity of the abilities they had 
identified. It was in the context of an overall plan to validate 
outcomes of college that we designed a set of parallel and 
interrelated studies to research abilities from multiple points 
of view, across multiple poin|.s in time, using multiple groups, 
with multiple opportunities for critique and comparison. These 
studies were then funded by a major three-year grant from the 
National Institute of Education toward the goal of establishing 
the validity of abilities learned in college for later careering 
and professional performance (Mentkowski & Doherty, 1977). The 
research studies, proposed in five project objectives for NIE, 
dealt with several themes. Or*e is related to identifying broad 
outcomes of college, including those abilities critical to 
effective performance at work, and how abilities can be defined, 
assessed and validat«^d. Another theme concerns the extent to 
which the curriculum contributes to development and change in 
outcomes, particularly if they are defined as 
cognitive- developmental patterns, learning styles and broad , 
generic abilities. A thiid theme relates abilities learned in 
college tc the world of work. 

We had a distinct advantage in designing and carrying out 
research on these issues. The faculty, with whom we were 
working, had already identified the more "intangible" outcomes of 
college such as life span development and lifelong, independent 
' learning, as important goals. They had spent several years 
identifying the broad, generic abilities they wanted theii. 
graduates to show (e.g . , communications , analysis , soc iai 
interaction, problem solving and valuing; Alverno College 
Faculty, 1976), and relating them in increasingly explicit terms 
to the program, courses and learning activities their students 
engaged in. These abilities were ,2fined as developing or 
teachable, as transferring acioss multiple settings and as 
internalized characteristics of the person, rather than discrete 
sets of skills. 



This gave us a full range of college-generated definitions to 
work with in researching student outcomes. The college's own 
methods for assessing each student's progressive development of 
her abilities (Alverno College Faculty, 1979) provided one set of 
measures for those outcomes. And we contributed to identifying 
and validating a set of cross-disciplinary measures of college 
performance (Alverno Col lege Assessment Committee/Office of 
Research and Evaluation, 1980, 1982, 1983; Friedman, Mentkowski, 
Deutsch, Shovar & Allen, 1982; Friedman, Mentkowski, Earley, 
Loacker & Diez, 1980). 
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Along with these definitions and measures, we identified a 
parallel set. These were drawn in part from other practitioners 
and researchers with wliom we were already working. While there 
were few, if any, measures that matched the faculty defined 
abilities directly, we selected measures representing the newer 
directions for defining and assessing broad, more intangible 
college outcomes, because these were most like the overall goals 
of the Alverno curriculum. Measures were selected that most 
nearly reflected the faculty's emerging theory of performance 
assessment. We administered a battery of twelve 
cognitive-developmental (Kohlberg, 1981b; Loevinger, 1976; Perry, 
1970; Piaget, 1972; Rest, 1979a), learning style (Kolb, 1983), 
and generic ability measures (Watson & Glaser, 1964; Winter, 
McClelland & Stewart, 1981) — we call them human potential 
measures — to over 750 students in a five year cross-sectional and 
longitudinal study. Two hundred of these formed the longitudinal 
group. Our goals were to describe change in college, to see if 
change could be attributed to performance in the curriculum, and 
to identify the underlying themes in these change patterns 
(Mentkowski & Strait, 1983). We atso thereby contribute to the 
development and further test of these measures (Mentkowski, 
Moeser & Strait, 1983). These twelve measures, along with five 
of our own that assessed student performance in the curriculum 
and enabled a test of its impact, yielded 17,500 responses. 

At the same time, we set about systematically gathering data 
about the students' perceptions. This meant creating an 
open-ended interview formiit that allowed students to generate 
their own definitions of the college experience, with particular 
emphasis on how they saw themselves changing, and why. We 
administered the interview to the same group of 80 students at 
the end of each year in college and to about 40 Seniors. These 
students were already part of the larger sample just described, 
and were completing the human potential measures in that 
longitudinal study (Much & Mentkowski, 1982). Altogether, they 
contributed nearly 400 interviews. 

To examine outcomes in the workplace and other post-college 
life settings^ we used several approaches. We first extended our 
interview studies beyond graduation. Over 30 two-year alumnae, 
also interviewed as Seniors, completed in-depth interviews where 
they discussed new learning at work, and the abilities and 
processes that enabled careering and professional performance 
after college. Second, we created a careering questionnaire for 
all 60 two-year alumnae. We were able to focus specifically on 
how new graduate and two-year aRimna attitudes and expectations 
evolve as they develop their professional roles and make career 
decisions since the same measure was concurrently administered to 
Seniors (Mentkowski, Much & Giencke-HoH , 1983). We initiated 
two studies with outstanding professionals in nursing and 
management (who were not Alverno alumnae), to derive models of 
the actual abilities these groups perform on the job, in order to 
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compare these with the outcomes sought by the college as well as 
those described by its graduates (Mentkowski, DeBack, Bishop, 
Allen & Blanton, 1980; Mentkowski, O'Brien, McEachern & Fowler, 
1982), Eighty nurses from three health care settings and over 
100 managers from 53 Milwaukee companies contributed performance 
interviews wnich generated 1,000 critical incidents. The 
prafessionais also completed careering histories, and ratings of 
abilities critical for education, selection and performance. 



We are finding some encouraging results: 

• the verifiable outcomes of a liberal education as taught 
at Alverno include broad, complex processes of the kind 
educators claioied, and these abilities can be reliably 
measured ; 

• they include cognitive-developmental patterns, learning^ 
styles, intellectual abilities and the more active/inter- 
active abilities sought in professional work situations, 
and abilities related to the broader domain of personal 
deve] opment ; 

• these intellectual and interpersonal abilities are the 
bafis of effective performance in professions; 

• student development of these abilities can be reliably 

-tributed to their successful performance in the 
curriculum, where students apply and use these abilities 
in coursework, simulations, external assessments and 
internships ; 

• both younger students, 18 to 22, and older more 
experienced students from varying life backgrounds 
show patterned development of these abilities; 

• students continue to develop these abilities and adapt 
them into personal and work settings both during and 
after college. Through the curriculum, students are, 
in essence, learning to learn. They continue personal 
and career development on their own; 

• these abilities can be related directly to those used 
on-the-job by alumnae and other effective professionals; 

• a validation design can be created that enables a college 
to demonstrate accountability to its constituents — by 
comparing its student and alumnae outcomes to the 
standards of the educational research community, 
professionals, and theorists of adult learning and 
growth. This effort improves the curriculum and 
contributes to general izable theory and practice of 
teaching , learning and assessment . 



We have also been able, along the wcy, to make some 
contributions to the developing field of educational program 
evaluation and to the repertoire of procedures for validating 
developmental outcomes. In many ways, we had to rethink the 
operaCional definition of validity as it is applied in a 
practiced-based research setting. Our work also seems to be 
offering some substantive support for the goals of 
outcome-centered curriculum design. 

These are early results from an effort that is now ongoing 
and a part of the learning process. But they do suggest that 
higher education can indeed help society achieve its goals for 
quality, equal' access and mobility by contributing demonstrably 
to each student's cognitive, interpersonal, and 

personal/ professional growth abil ities. They indie <ite that 
Alverno's curriculum also contributes to the student's ab-.lity to 
int>egiatt these abilities and apply them effectively in later 
life settings, particularly in the world of work. 

An ability-focused curriculum with a strong emphasis on 
assessment and integrated learning across an institution can 
satisfy student needs for personal growth and career. 
Ability-based learning yields both traditional liberal learning 
values and the high performance learning outcomes that prepare 
students for their place in work and society. 
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WHY FOCUS ON OUTCOMES IN HIGHER EDUCATION? 



Why specify outcomes in higher education? Society as a whole 
is currently questioning if outcomes claimed by higher education, 
especially the liberal arts, are actual ly achieved . This is 
partly because societal needs for higher education have changed. 
We have become a service rather than a production society, where 
more and more of us need a quality education and technological 
skills. We have become a knowledge society, where the emphasis 
is on using knowledge because we can no longer master it all. 
And we have become a society of rapid change, where each person 
needs preparation for changing jobs and responsibilities. 
Employers of college graduates complain that graduates no longer 
have traditional outcomes of college such as thinking, writing, 
and problem solving, let alone the ability to adapt skills to 
changing roles and contexts. 

Periods of economic stress sharpen the demand for usefulness. 
There is more emphasis on showing that abilities learned ir 
college make a difference in contributing to society after 
college. Consequently, higher education is expected to show a 
relationship between abil ities learned in col lege and 
professional productivity and development. Education for more 
productive work has become a new theme on college campuses. 

In the past, college as preparation for life was generally 
assumed. Highly selective colleges admitted persons with high 
scores on admissions tests ^id were rarely asked to demonstrate 
that their graduates had productive lives after college because 
stud ies of col lege outcomes showed that income , status , and 
productivity in the work force were more pronounced among college 
graduates. The new student body has changed that. Minority 
ethnic and racial groups, the poor, the handicapped, and women 
are now making up a larger segment of the college population. 
They are coming to coll eg e expecting higher status jobs . 
Economic and social mobility are thought to result from higher 
education. Minorities expect that college will assist in erasing 
discrimination and allow them greater access to society's 
benefits. Nontraditional students, adults who are already 
experienced in multiple roles, are also coming to college in 
record numbers. They expect that college learning does indeed 
build on life and work experience, and is not just a paper 
qualification. Thi*^ is in sharp contrast to an outmoded concept 
of college as an opportunity to momentarily escape from life's 
pressing demands, and to experience learning for its pwn sake, 
unencumbered by the need to earn a living or uO support a family. 
Yet both younger and older students look to college for 
assistance in their search for meaning in a confusing world. 

Expecting that abilities learned in college will directly 
contribute to one's opportunities and success at work comes 
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particularly from the large nontraditional college population. 
This group, particularly women, have already experienced the 
impact of lack of abilities that are needed for advancement out 
of traditional clerical and service positions to positions with 
more responsibility and opportunity. These women are lively to 
expect, ask for and demand that college be accountable to 
demonstrating that the educational outcomes taught are those 
necessary to achieve their own professional goals (Cross, 1981). 
And traditional age students are now joined in this expectation 
of a Career after college (Astin, 1982). 

Higher education faculty question higher education's ability 
to respond to these needs. They ask if liberal arts outcomes can 
survive in the new aura of learning for work rather than learning 
for its own sake. Can liberal arts goals be developed in an 
atmosphere of professional education and* education for work? 
Will open access lower standards? Will the more traditional 
outcomes of college be sacrificed for graduates' technical 
expertise? Are students still learning to analyze, to think 
critically, to solve problems, create new ideas and ways of 
thinking, to appreciate multiplicity in context and culture, and 
to achieve quality of life? Can colleges be responsive to the 
new student body and the values of today's student and 3till 
maintain high standards for student performance in college? 

Faculty also question how liberal arts colleges can maintain 
an orientation to the demands of society to teach toward 
careering and the needs of the marketplace and still maintain the 
"student centered**^ atmosphere of the liberal arts college. Here 
student development is a primary outcome and focus. College is a 
time to find one's way out of adolescence and to take on adult 
responsibilities, or to broaden one's world view through the arts 
and humanities. 



College students in general are also pressing for the more 
intangible outcomes of college. Self^ful fillment has been 
labeled as the "new morality" in our society (Yankelovich, 1981) 
and college students are also expecting their efforts to bring 
self- fulfillment and personal development. While 

self- fulfillment is clearly a goal, college students also expect 
advancement and career achievement (Ast in , 1982) . Minority 
groups and women have also come to appreciate the insights 
college offers for developing their role as active members of 
society. 

All of society seems to be more interested in accountability. 
The consumer movement, the rise of special interest groups, are 
two indications Lhat individuals are expecting institutions to be 
more accountable, to complete their share of the contract. 
Colleges have been known to promise economic and social mobility, 



personal growth, and other broad outcomes. Students are more and 
mor* taking them at their word, and expecting that the degree 
makes a difference. They are asking colleges to demonstrate, not 
just to promise, that the insititution will be responsible for 
fulfilling their part of the bargain. Expectations for quality 
of life, for careering after college, fo. preparation for life as 
well as work, for achieving personal development as well as 
professional development, all create an atmosphere of 
accountabil ity . 

Finally, there is the current quest for quality. Quality in 
education is now deiPanded by students, higher education faculty, 
commissions on excellence, and by society at large. These 
expectations for quality are there despite accelerated change, an 
infci^ftdtion and technological explosion, and needs for equal 
access and equality of educational opportunity. For many, 
accounting for quality means demonstrating relationships between 
college, personal growth and professional development. All of 
these reasons have prompted the move toward defining, assessing, 
and validating educational outcomes. 
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WHY VALIDATE OUTCOMES IN HIGHER EDUCATION? 



We have just outlined the rationale for focusing on outcomes 
m higher education. Why establish the validity of outcomes? 
Most colleges and universities have not felt compelled to 
validate that students achieve outcomes or to relate those 
outcomes to future outcomes. Indeed, the effectiveness of 
college has often been taken for granted. Why launch a major 
effort to validate outcomes? 

The rationale for validating outcomes is similar to that for 
focusing on outcomes. First, the press for accountability in 
higher education is logically translated into demonstrating that 
education is related to and is adequate preparation for vork, and 
that education is adequate preparation for life. Validating the 
outcomes of college means demonstrating that a liberal arts 
education assists students to meet the prerequisites for later 
personal and professional performance. But the press for 
accountability is not just a utilitarian one. We are no longer 
interested in demonstrating only that education is useful. We 
are interested in demonstrating that education is equitable, that 
persons without traditional backgrounds can achieve traditional 
outcomes. Demonstrating that our open access policv does nor 
lower quality is as important as being accountable. 

Still more important for the adi'lt student is the need to 
show that outcomes achieved can be attributed to the college 
experience rather than just to maturation. Does college enhance 
life experience for the older adult, or does education interfere 
with, rather than build on experience? We are no longer willing 
to accept that outcomes demonstrated at graduation are valid 
unless they persist over time, or contribute in some way to the 
development of later abilities that are critical to future 
outcomes (Astin, 1977, p. 210). 

If outcomes are no longer defined as static, but as 
developmental, then change and its causes are important aspects 
of demonstrating ^-alidity. What curricular aspects cause change 
in higher education? This question shows an increased emphasis 
on the importance of continued program development. It is 
generally recognized that embarking on validation research can 
enhance higher education's ability to create effective 
prog**amming . Focusing on abilities and processes as outcomes, 
rather than knowledge alone > and assessing for them in a 
per formance-based curriculum, is a "new idea" in higher 
education. Consequently, it is expected to prove itself — to show 
that it is doing what it claims to do. New strategies are 
usually much more open to question and expected to be researched 
before adept: ion. 
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Validation research can do much to enlighten us on the 
characteristics of the new student's learning, abilities, 
cognitive-developmental patterns and learning styles so that all 
of higher education, whether or not it is performance-based, can 
become more responsive to student needs. How do students learn? 
How do they develop? And how do students actually experience 
learning — from their point of view? Validation resea. h i*. 
critical to building a general izable educational model tor adult 
learning and development, particularly in view of the needs of 
today's more nontraditional student body. 

Finally, validation research that identifies the abilities of 
effective pro fess ionals bridges the gap between the col lege 
faculty and the professional community. Both groups have a stake 
in ensuring that abilities learned in professional programs are 
those critical for effective performance at work after college. 
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DEFINING, ASSESSING AND VALIDATING COLLEGE OUTCOMES 



In response to recent concern about the value of a liberal 
arts degree, college educators are beginning to identify, measure 
and credential broad abilities that are expected outcomes of 
college (Loacker & Palola, 1981)^ Moreover, some college 
educators are no longer satisfied to judge program effectiveness 
by comparing their students* performance against standardized 
test norms. Rather, they are questioning how colleges might 
assess students using criteria or standards derived from outcomes 
describing the performance of a liberally educated, competent 
adult. Other educators view college as a catalyst for lifelong 
development, and want to know if abilities learned in college are 
related to the future personal and professional performance of 
graduates (Mentkowski & Doherty, 1977). 

These educators are interested in comparing students* mastery 
of broad abil ities to their potential for enhanced human 
development. How do outcomes characteristic of colle<^e students 
compare with their developmental otential, with what is possible 
for them to achieve as humans? Some educators feel these 
questions should be raised not only about learned abilities 
faculty can currently measure and' credential, but also about the 
more "intangible" outcomes of the college experience, those 
traditionally promised to graduates by most liberal arts 
colleges. These more intangible outcomes include continued 
life-span development, transition to "life after college," 
transfer of learning to various settings and professional 
positions, self-directed and integrated personal functioning and 
1 if e long learning . 



College Outcomes: Frameworks and Measu res 

I Educators are beginning to define and assess for broad 

generic abilities or competences, and more intangible outcomes. 
I Their goal is to further define and understand the nature of 

I abilities and outcomes they teach toward as an important source 

" for curriculum development. One problem these educators face is 

the lack of standardized external criterion measures that measure 
I abilities and that predict later performance after college, to 

I which they can compare stuu^iii. performance outcomes. There has 

been more interest in operational understanding of broad outcomes 
I since publication of The American College (Sanford, 1962), and 

I . the recent move toward outcome-centered curricula is a thrust in 

tha direction (Grant & Associates, 1979). 

I In the recent past, some educators, colleges and professional 

schools have identified outcomes and developed ways to assess 

I them (Grant & Associates, 1979; Loacker, 1981). (Examples 

I include Alverno College, AnCioch School of Law, Brighara Young, 

College III of the University of Massachusetts, College for Human 

I Services, Delaware County Community College, Florida State, 
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Harvard University, Iowa Wesleyan, Kirkhof College, Mary College, 
Metropolitan State, Miami-Dade Community College, Mt . Hood School 
of Nursing, New Rochelle College, North Adams State, Northwestern 
University School of Music, Our Lady of the Lake, Southern 
Illinois University School of Medicine, University College at the 
University of Louisville^ University of Montana School of Law, 
the University of New Mexico School of Medicine, and others.) 
iiany of these institutions are now addressing outco'^i validation 
issues. They are asking hard questions about the extent to which 
students are able to demonstrate outcomes educators have 
identified as important for all college students to master. But 
what measures ate available that will contribute to program 
evaluation and outcome validation? 

Linking Education and Work: 
Generic Ability Measures 

Several efforts in defining and assessing college outcomes 
ar J specif ical ly focused on performance measures of general 
^^-*V^M.lities and characteristics predictive of effectiveness in 
Vi^ter life (e.g., ACT's College Outcome Measures Project, McBer 
and Company's Cognitive Competence Assessment Battery). These 
more focused measures might appear redundant with the usual grade 
reports and standardized achievement or aptitude tests in 
predicting future performance. Yet these conventional measures 
and indices have not shown much relationship to later behavior 
(McClelland, 1973, 1980). The effectiveness of the new 
performance measures has not been deterjiined as yet, but initial 
tests are underway in this study and elsewhere (Winter, 1979; 
Winter, McClelland & Stewart, 1981), 



In 1975, the Fund for the Improvement of Post-secondary 
Education supported a consortium of colleges in trying out some 
newer measures to assess outcomes. As a member of this group of 
colleges, Alverno participated in the FIPSE project, awarded to 
McBer and Company, by administering some of these new measures. 
These instruments, collected or developed by McBer, later became 
known as the Cognitive Competence Assessment Battery (Winter, 
McClelland & Stewart, 1981). 



When Alverno sought to identify external criterion measures 
for inclusion in a validation study of student outcomes, we 
selected these measures because they most nearly represented some 
of the abilities identified by Alverno faculty. The Cognitive 
Competence Assessment Battery provided a particular focus on 
generic abilities of analysis, and included assessment of motive 
dispositions ano other characteristics important to the 
relationship between learning and later behavior. Because other 
colleges were also administering these measures, we could count 
on some comparison data. 
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These newly-developed measures of generic abilities can serve 
as better outcome measures, but we are still faced with the need 
to measure abilities learned in college in the context of 
lifelong learning and development. How are abilities learned in 
college transformed through personal and professional experience? 
How can we recognize them in the older adult? The search is on 
for better ways to measure the more intangible outcor^e? of 
college, those that are often referred to as personal development 
outcomes (Bray, Campbell & Grant, 1974) or other personal 
maturity variables (Heath, 1974, 1976, 1977). How else might we 
ensure that college outcomes become integrated aspects of the 
whole person that might be expected to develop^eyond college? 



It is in relation to the problem of defining and assessing 
abilities learned in college set within a context of lifelong 
learning and life-span development that we proposed using 
cognitive-developmental theorists' descriptions of human growLh 
and development as sources for college outcome measures 
(Mentkowski & Doherty, 1977). Indeed, Chickering and Associates 
(1981) have recently advanced the argument that an overarching 
goal of higher education is encouraging developmental change. 
Developmental psychologists have described broad developmental 
domains that can be measured, such as moral development 
(Kohlberg, 1976); ago development (Loevinger, 1976); cognitive 
development (Piaget, 1972); and intellectual and ethical 
development (Perry, 1970, 1981). These theorists provide us with 
descriptions of the way in which individuals cognitively 
structure meaning and make sense out of their experiences. 
Descriptions of development, whether via a series of stages 
(piaget, Kohlberg), ego levels (Loevinger), or positions (Perry) 
provide us with a partial picture of . .students' potential for 
growth. They describe some of the more universal outcomes of 
human functioning against which educators can validate more 
intangible curriculum outcomes. .-^ 

While we do not expect that educators will use a student's 
current developmental level, position or stage as a measure of 
performance to credential or pass a student, such information can 
be used to describe where the student is in his or her 
development . Assessing student performance on these measures 
over time givti us important information on individual patterns 
of development during college, and helps us evaluate the extent 
to which college ot specific curriculum interventions are 
contributing to the general cognitive growth of learners. 

This approach to validating student outcomes suggests 
assessing students on various levels of cognitive development as 
part of program evaluation designs. Using 



Deveiopmentai Theory: 
Cognttive-Deveiopmerital Measures 




cognitive-developmental measures to assess college outcomes has 
another important value. The results can be used to inform 
instruction, and to assist in creating appropriate curricula. We 
expect this research co reduce the "size of the existing gap 
between developmental theory and educational practice" (Astin, 
1983). 

Experiential Learning Theory: 
Learning Style Measures 

Experiential learning theory and research has more recently 
described learning ar a process (rather than as static outcomes), 
where knowledge is cheated and linked to action through the 
transformation ol experience (Kcib, 1983). While 
cognitive-developmental theories a.-ii^ixbe assimilation and 
accomodation as the basis for an- interactive ^earning process, • 
these theories are less likely to describe individ\xal differences 
in learning. Cognitive-developmental: .patterns tend to describe 
common paths in the growth of intellectual development. A 
variety of reseachers hive centered on learning style as an 
important indicator cf student learning and development (Curry, 
1983). Basically, the^e researchers are interested in specifying 
individual differences in apprbaches to learning, cognitive 
styles, and differences in learning style preferences. Since 
feedback on learning style is one way to assist students to 
analyze their own approaches to learning, faculty find that 
learning style measures can be important nat only for curriculum 
design, b||t also for assisting students to become more open to 
other modes of learning (Deutsch & Guinn, Note 1). The Council 
for the Advancement of Experiential Learning has supported 
development of teaching and assessment strategies based on 
learning by experience (Keeton t-vTati^, 1978); and giving credit 
for learning that occurs in other '"-'ttTarT^ formal , or classroom 
Irearnlng settings. Experiential learning is seen as a process 
that links education, work and personal development (Kolb, 1983). 
We have proposed using learning style measures as a way to tap 
college outcomes particularly because Alverno's curriculum is 
based partly on experienti^^L-learning theory (Doheity, Mentkowski 
& Conrad, 1978), and because of the strong emphasis on student 
involvement in both in-class and off-campus learning experiences 
(Mentkowski, O'Brien, Cleve & Wutzdorff, 1983). 

J- 

Competence Assessment: 
Performance interviews and Inventories 

Another approach to the definition and assessment of outcomes 
we researched in the current studies was the performance 
assessment of effective professionals in order to build modelr of 
their abilities or competences. While performance assessment of 
alumnae is rare, we determined it to be a way to identify 
abilities alumnae do perform after college, to establish a link 
to abilities learned during college. Performance assessment of 
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alumnae was beyond the scope of the current set of studies until 
we had first completed the round of open-ended perspectives 
lnt3rvievs and careering questionnaire's (see below). We do plan 
future alumnae studies using performance interviews. For the 
current study, we did use performance interviews to assess the 
competences of outstanding professionals . In addition , we 
employed performance characteristics inventories which enable a 
study of professional perceptions of the abilities, competences 
and behaviors descriptive nf outstanding versus average 
perfotmers. We selected the approach of Job Competence 
Assessment developed by McBer and Company (Klemp, 1978; 
McClelland, 1976) to build professional competence models, 
because the underlyiiig definition of abilities or competences and 
principles of assessment most nearly matched that of the Alverno 
faculty. 

Perspectives on Learning and Careering: 
Interviews and Careering Questionnaires 

The outcomes of college also need to be described from the 
student's perspective. Clearly, development of college outcomes 
measures focused on abil ities acquired during col lege and 
expected to be related to performance after _ col 1 ege , that 
describe intellectual and personal growth across the life span, 
and pex.formance assessment of professionals on-the-job, is just 
getting underway. Measures of cognitive^developmental patterns 
have been used primarily for z:esearch purposes, and measures of 
learning styles, while many and varied, have little experience as 
college outcomes measures. 



It seemed imperative, then, to take a path initiated by Perry 
(1970) in the sixties, that of conducting open--ended interviews 
to discover how students experience college. We proposed 
conducting broad, in-depth longitudinal interviews with stud^'^.ts 
to tap their perceptions, and to thereby gain some insight .nto 
the determinants of the outcomes of college from the student's 
point of view. We a Loo expected to uncover some of the 
individual differences in learning patterns and the several paths 
that students take during college to achieve their goals. We 
hope to expand our understanding of wh^ benefits from college and 
why, and what kinds of {experiences, characterize students in a 
performance-based or outcome-centered liberal arts curriculum. 
Further, the interviews could serve as a nontext for interpreting 
results froQ the human potential measures, and for seeking the 
links between abilities learned in college to those demonstrated 
after college. W!. le some of these research goals go beyond 
those reported hets: this approach is effective in raising 
further research hypotheses and for communicating the nature of 
student change to faculty. 
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We developed careering questionnaires to assess the students, 
alumnae and other professionals. These questionnaires allow 
collection of demographic data, information on paid and unpaid 
employment, careering history, and attitudinal information. 
Careering questionnaires also collect data on a range of 
variables that provide a context for the performance and 
perception studies of professionals* 

Matching Franieworks and Measures to Curricular Goals 
and Assessment Principles 

\ primary reason for undertaking evaluation and validation 
studies of student outcomes in college is to inform continued 
curriculum development. This includes icore clearly specifying 
outcomes , learning strategies, assessment techniques and 
evaluation methods. Educators are working to develop curricula 
that respond to the students' learning styles, that capitalize on 
the adult's range of experiences and that reflect what is 
und'srstood so far about patterns of younger and older adult 
development and learning. But this effort will succeed only if 
we question the selection and effectiveness of current frameworks 
and corresponding college outcomes measures for college 
curricular settings . 

Clearly, selection of frameworks, and corresponding 
instruments as external criteria or standards against which a 
college examines its ability to facilitate student growth is 
appropriate if there is: (1) a match between the goals and 
obiectives of the college and the framework used, and (2) a match 
between the college's principles of assessment and the theory of 
assessment used to develop instrumentation based on the 
framework: 

Instruments which have been used for theory testing — even 
though they have demonstrated reliability and validity— need 
to be filtered first through the practitioner's goals^ 
objectives, learning strategies and assessment processes. 
Once they emerge from this crucial dialectic, they may be 
effective program evaluation instruments well' (Mentkowski, 
1980, p,28). 

to 

Therefore, our practice-oased research using any of the 
measures to establish the validity of college outcomes needs to 
be understood in the context of their use. This context at 
Alverno College includes a philosophy of education, an 
outcome-centered curriculum and principles of assessment which 
have been in the process of development by Al^'ernf> faculty for 
over ten years (Alverno College Faculty, 1976, 197.^ 1979). 

28 
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DEFINING AND ASSESSING OUTCOMES AT ALVERNO COLLEGE 



^Iqw Do We Define Outcomes ? 

Alverno's faculty are concerned with defining and assessing 
outcomes of college. The student's continual development is at 
the center of institutional goals. Thus, the major outcome of 
college is growth or change. Faculty expect college to be a 
significant and positive facilitator for student growth, and a 
catalyst for lifelong learning and development. Rather than 
thinking of college as a cause and student growth as an effect, 
growth is a result of the interaction between a self-directing 
individual wno plays a role in initiating her own growth, and a 
learning process. Both faculty and student select and involve 
her in learning which challenges and supports personal change. 
The role as learner continues after college throughout the life 
span, and learning becomes a means by which she realizes, her 
potential for professional development and personal growth. 

This emphasis on growth of the person across the life span, 
for which college is a catalyst, determines what broad outcomes 
are identified. Yet any definitions of outcomes need to retain 
the breadth and complexity characterized by college-level 
learning and performance. The college takes responsibility for 
contributing to growth and development of lifelong learners, and 
for learning in college that continues after college. Such goals 
are broad, and a commitment to them provides a philosophical base 
for a faculty working collaboratively to develop a curriculum. 
But ultimately, a faculty needs to define these broad, more 
"intangible" outcomes of college if they are to teach and assess 
for them. 



What Are the Abilities or Competences ? 

What are the developmental, holistic and generic abilities 
each student must demonstrate in order that faculty consider her 
a lifelong learner? At Alverno, the focus on outcomes took shape 
ill 1971 when the faculty, in a yearlong series of intense faculty 
institutes, struggled to respond to the questions, "What should a 
student get out of spending four, years with us?", "What kind of 
person did we hope she would become?" and "How are we helping it 
to happen?'* As the year progressed, it became clear that a focus 
on outcomes a liberal education challenges the individual to 
develop, needed to be companied with questions about the 
definition of abilities, the nature of the learning experiences 
provided , and the way in which abilities — we called them 
competences — could be assessed (Alverno College Faculty, 1976). 
For the next two years, an academic task force synthesized the 
many abilities the. faculty identified into eight general outcomes 
and defined each as an ability or competence. Each was then 
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further defined via a sequential, increasingly complex set of six 
levels. The competences are: 

• Communications 

• Analysis 

• Problem Sr^lving 

• Valuing 

• Social Interaction 

• Taking Responsibility for the Environment 

• Involvement in the Contemporary World 

• Aesthetic Response 

All students are expected to progressively demonstrate levels 1 
to 4 of each ability, usually by the end of the general education 
sequence • She then demonstrates levels 5 and 6 of those 
abilities most related to her major and minor areas of 
concentration. 



Faculty have defined the meaning of each ability or 
competence, the sequence and increasing complexity of the 
competence levels, the relationship of each competence level to 
other levels aad to other competences as well as the 
relationships across academic disciplines in the Facultjr Handbook 
o n Learning and Assessment (Alverno College Faculty, 1977) . 
Instructional methods are suggested • Each competence level also 
describes the criteria for assessment, and suggests appropriate 
instrument stimuli and modes (with an emphasis on production 
tasks) for assessing performance. At Alverno, college outcomes 
are defined as abilities or competences considered to be complex 
processes. Faculty define abilities as developmenta l , holistic 
and generic (Alverno College Faculty, 1979) . 



Developmental Abilitief 

For an ability or competence to be developmental medns that 
it is teachable. Thus, the ability or competence is broken open 
into sequential descriptions or pedagogical levels that describe 
increasingly complex elements and/or processes which are acquired 
by students over time as the result of instruction and where each 
level requires a more complex demonstration of the ability. 
Further, competences that are developmental continue to change 
after college, as additional learning experiences contribute 
toward developing greater complexity. 

Holistic Abilities 

For an ability to be holistic means that each developing 
ability involves the whole person. Complex abilities or 
competences include a behavioral component, a knowledge 
component, an affective or self-perception component, as well as 
a motivation or disposition component (Klempi 1979). All or most 



ERIC 



20 30 



of the components of a competence or ability can be inferred from 

an observable demonstration or performance. Traditionally, 

colleges have required demonstration of only the knowledge • 

component. When competences or abilities are defined 

holistically, then knowledge, skill, attitudes, self-perception 

and dispositional components are specified. Within a particular 

context, abilities or competences can then be defined as 

observable behaviors. These components are expected to become 

integrated, and together with other abilities, involve the whole 

person. ^ 



Generic Abilities 



For an ability or competence to be generic means that the 
developing, holistic ability will transfer across situations and 
settings. Thus, abilinies are defined as tran :f erable . The 
kinds of situations to which abilities are expected to transfer 
include those a student encounters in exercising multiple roles. 
Generic Abilities are expected to transfer not only to situations 
in college and work, but also to personal and professional 
situations after college. Generic abilities equip students with 
skills that transfer from one situation to another, across roles 
and positions within a particular occupation, and even across 
occupations. 

Most students will ultimately be taking on different roles 
simultaneously. The abilities they acquire in college are 
expected to assist them not only in their professional roles, but 
in personal roles such as citizen, family member and parent afcer 
college. Professional roles, as well as the personal ones, 
continue to change and develop. Acquiring abilities that are 
developmental, holistic and generic assumes that students become 
learners in college and become self-directed in learning how to 
learn. Learning how to learn consists of learning Jtrategies 
that make up the concept of "lifelong learner." We expact that 
our studies of student and alumna perspectives on learning and 
careering will help us define these broad concepts. 

How Can We Develop These Abilitiei ? 

How can faculty develop these abilities in each student so 
abilities become internalized, integrated and general i^able? In 
1973, the faculty implemented an "outcome-centered" curriculum 
and developed learning methods to ter:h toward the competences 
(Alverno College Faculty, 1977). The curriculum emphasized 
assisting the student to develop these abilities in Ways that are 
unique to her own individual differences in learning style and 
how she conceptualizes learning. Learning strategies build on 
the theory of "experiential learning" (Doherty, Mentkowski & 
Conrad, 1978). The experiential dimensions of the curriculum 
have been expanded such that students in each of the 18 academic 
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and professional departments are immersed in opportunities to 
experience the constraints of the work world by engaging in 
mentored off-campus experiential learning (OCEL) where 
transferring abilities learned in college is paramount. 
Classroom learning experiences likewise focus on student 
involvement in learning situations where concrete experiences, 
reflection, conceptualizing ideas and concepts, and plans for 
action are tested out in new performance situations (Mentkowski , 
O'Brien, Cleve & Wutzdorff, 1983). 



How will faculty know a student has achieved these abilities 
or competences according to their prescribed criteria or 
standards? The assessment process developed by the Alverno 
faculty has been described elsewhere (Alverno College Faculty, 
1979), and represents one of the more recent directions in 
reconceptualizing assessment (Willingham, 1980). The assessment 
process is patterned in part on assessment center technology 
(Moses & Byham, 1977). Alverno relies on volunteer assessors 
from the Milwaukee business and professional community to judge 
effective student performance, as well as the faculty who design 
instruments and judge performance both in the Assessment Center 
and through cour&^s. 

Four fundamental principles of assessment are spec if ying 
criteria , relying on multiple judgments , choosing from alternate 
performance modes so as to elicit the ful 1 range of the 
developing ability, using expert judgment to infer student 
abilities from this performance, and providing the student aa 
opportunity for self-assessment . 



Once outcomes are defined as abilities or competences, 
assessing for them means defining the criteria for judging 
student performance. Thus, faculty have defined abilities or 
competences not only by the competence levels, but also by 
specifying assessment criteria. 

An important characteristic of assessment is that of 
evaluacion of student performance in relation to criteria or 
standards (criterion-referenced), in contrast to . students 
performing relative to norms (norm-referenced) created just from 
the range of performance of other students. Vftiile standards are 
informed by the tange of student performance, they are also open 
to input from other Sources (e.g., dviscriptions of abilities or 
cognitive patterns from theories of learning and development; 
abilities that characterize effective professional performance). 



How Will We Know a Student 
Has Achieved These Abilities? 



Criteria 



22 




ERIC 



4 



ERIC 



Identifying appropiate criteria or standards is a difficult task 
and worth a research agenda of its own (Glaser, 1981). The 
results reported in this paper are intended to assist faculty 
with this task. 

An issue that arises when specifying criteria is the 
relationship of the criteria to the abilities one is measuring, 
and al so the relationship of those abil it ies to broad and 
inclusive college outcomes such as "lifelong learning." "reaching 
one's full potential," "becoming an independent learner," 
"developing critical thinking" and "learning to learn." 
Abilities students must perform in order to graduate, as defined 
through assessment criteria, can be distinguished from broad 
outcomes that are more intangible. Educators may agree on these 
more intangible outcomes and may consc iously xiae them as 
frameworks in teaching. Tney may even assess for them 
diagnostically in many ways . And faculty have used these 
outcomes to select external criterion measures to validate the 
outcomes of college. But educators do not demand evidence from 
student performance assessments in order to graduate students, 
tior do faculty guarantee such outcomes. 

Specifying criteria for assessment is a faculty effort to 
make the more intangible outcomes of college, and defined 
abilities or competences, operational. Faculty work to identify 
both specific and broad criteria for judging student performance 
at a particular competence level. For each broad ability to be 
assessed, faculty must make the ability explicit through criteria 
so students can understand what per formance is required. 
Therefore, faculty need to describe the ability sufficiently 
through criteria statements such that it can be reliably and 
validly assessed. At the same time, the complexity of the 
abilities assessed limits how explicitly these criteria are 
stated. Criteria for assessing student performance of abilities 
fall on a continuum from broad to specific. Thus, assessment 
calls for multiple, expert judgment by faculty. 



Alverno faculty also recognize that any one sample of student 
performance is just that — a sample of what the student is able to 
do in a given context, in response to a particular instrument 
stimulus. Consequently, Alverno faculty rely on multiple 
judgments. This means observing her performance cumulatively, in 
a number of contexts, across a number of settings, across time, 
and across a variety of performance modes. 




Alternate Performance Modes 



An important challenge in defining criteria for assessment is 
to require that students demonstrate not only the knowledge 
component of abilities, but also demonstrate the behavioral, 
dispositional and self-perception components. Learning to do as 
well as to know puts the emphasis on learning, how to perform, and 
requires that the performance- mode match, as nearly as possible, 
the ability being assessed. 



Because of the complexity of the competences being assessed, 
faculty design instruments complete with stimulus and performance 
mode (and criteria) that elicit to the fullest extent, the 
student's developing ability. Thus, Alverno faculty have 
committed themselves to designing assessment techniques that 
employ production tasks rather than recognition tasks. That is, 
the student is required to generate a response to an instrument's 
stimulus, rather than simply to indicate recognition of 
information. Consequently, faculty are likely to employ 
performance modes such as essay, group discussion, oral 
presentation, interview, and in-basket, rather than modes such as 
multip^le choice, short answer, true-false, etc. Performance 
modes are designed requiring the student to demonstrate behavior 
similar to the ability as usually expressed rather than an 
artificial mode (e.g., to demonstrate Social Interaction skills, 
she would perform in an actual group discussion). 



Expert Judgment 

Use of production tasks requires expert judgment, defined as 
special knowledge or skill ("expertise") that the assessor brings 
to the judging situatioti and applies in a rigorous or disciplined 
way. In the context of higher education, where faculty teach 
toward sophisticated abilities, complex cognitive structures, and 
highly skilled performances, faculty are accustomed to tha use of 
expert judgment in instruction and assessment. Expert judgment, 
which involves the use of inference in abstract analytical 
thinking, is basic to assessing student performance at advanced 
levels. Expert judgment is a practical instructional and 
assessment tool and is in constant use by faculty in higher 
education who insist on production tasks to assess performance. 
A treatment of issues surrounding the use of expert judgment can 
be found in Mentkowski, Moeser and Strait (1983), 

Self-Assessment 



Self-assessment, or student assessment of her own 
performance, her perceptions of the extent to which her 
performance meets criteria, is an important component of the 
assessment process. Assessment provides a challenge that assists 
the student to take responsibility for her own learning, to 
assess herself, and to become more self-directed. 




I Assessment of student performance leads to evaluation and 

revision of instruments and clarification and further development 
of criteria for assessment. Faculty work to continually clarify 
I and develop criteria so as to specify both specific and generic 

i criteria for credentialing student performance. 

I These characteristics of assessment are important to 

recognize because they have implications for the selection of 
I external criterion measures for validating the faculty defined 

I outcomes of college, and for realizing our project objective to 

validate Alverno assessment techniques. 

I What Are Student Outcomes of the Learning Process ? 

I Since outcomes are very generally defined as growth or 

I change, and are visible as change in performance , ability or 

competence definitions communicate what the student does or 
performf., rather than what the faculty does or performs. Note 
I that college outcomes include self-assessment, or change in the 

' studenv's perception of herself as a learner and as a growing, 

changing individual* In addition to student performance, student 
I perceptions are equally valuable outcomes of college. 

I What Are Alumnae Future Outcomes ? 

Because faculty define college outcomes in relation to the 

student as lifelong learner, faculty also seek to define future 
I outcomes , to attempt to "see" and conceptualize outcomes that 

I develop from those demonstrated in college. Future outcomes help 

provide a picture of abilities as they appear "full grown." They 
I orient faculty toward defining outcomes of college in ways that 

I describe the beginning of abilities as they are taught in 

college, in relation to those graduates will need five, ten or 
^ even twenty years after college. Abilities needed for the future 

I are built on abilities taught in college. Analytical thinking 

' expressed by deriving a hypothesis from a set of interrelated 

studies for a biology class may be quite different from the 
I inductive, problem finding analysis an environmental speci^list 

I uses on the job. College must educate students for the future, 

not just fcr the present. Analytical thinking defined for 
I college learning must be related to post-college roles to ensure 

I future personal and professional outcomes. Yet we know very 

little about what those relationships are. 

I Future outcomes also include student expectations for 

realizing career and pro-fessional opportunities, expectations 

I that an investment in college wil 1 contribute to adequate 

I preparation for performing in professional situations , 

realization of self-fulfillment, and an enhanced quality of life. 

I Beyond student expectations, faculty expectations for students 
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include an expanded role as a learner who can make the transition 
from college to work and to life after college. Faculty want 
students to become self-directed learners and to work toward 
achieving personal and professional goal integration (Earley, 
Mentkcwski & Schafer, 1980)- 



What Are the Components of a 
Learning Process? 

The six questions underscored above are repeated in Figure 1, 
a graphic of faculty questions and learning process components. 
They set the stage for the development, in 1976, of the ^acuity 
focus on establishing the validity of the outcomeb of College. 



LEARNING PROCESS 



What are the developmental, holistic 
and generic abilities each student 
must demonstrate in order that we 
consider her a lifelong learner? 



How can we develop these abilities 
in each student so they become 
internalized, integrated and 
generalizable? 




COMPETENCES 



EXPERIENTIAL 
LEARNING 



How will we know if each student 
has achieved these abilities according 
to our prescribed standards? 



ASSESSMENT 

PROCESS, 
TECHNIQUES 



What are the outcomes of the 
learning process, those credentialed 
and those expected but not 
credentialed? 




STUDENT 
CHANGES IN OUTCOMES 



What are alumnae realizations in 
perception, careering and profes- 
sional performance as a lifelong 
learner? 




ALUMNAE 
FUTURE OUTCOMES 



Figure 1. A description of Alvemo learning process components. 
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Establishing the validity of outcomes 

Validation studies can be an important source for insight 
about how human beings learn and develop. Educators are urgently 
seeking the best available frameworks for understanding what and 
how their varied students learn, which experiences stimulate and 
enhance that learning, and how that learning fits into the tasks 
of lifelong growth* Such studies are also designed f'^r 
verification demanded by the need for accountability. Basically, 
validation helps to focus four kinds of questions which are asked 
by educators as well as by the constitutencies they serve: 

• Descriptive questions: "What is occurring?" 
"How is it occurring?" 

• Ascriptive questions: "Why is it occurring?" 

• Evaluative questions: "Is what is occurring 
'good' compared to a criterion or standard?" 
"Is the standard valid?" 

• Prescriptive questions: "What should be 
occurring?" ^ 

As t^e new science of program evaluation has emerged, it has 
become apparent that existing resources for establishing validity 
(e.g.) American Psychological Association, 1974) are not 
sufficient to the task of validating developmental outcomes. Nor 
is the control led-experiment model on which these approaches are 
predicated either appropriate or possible in a dynamic, 
interrelated practice setting (Bryk, 1983; Cronbach & Associates, 
1980; Parlett & Hamilton, 1976). 



Like several other investigators (Grant, Note 2; Messick, 
1980; Popham, 1978), we have therefore opted for a validation 
approach geared to the unusual complexity of the learning 
outcomes involved in college, as well as to the fluidity of 
program and population that characterize college instruction. 
Several features represent our attempt to respond more 
effectively to the constraints and opportunities of validating 
developmental outcomes in a dynamic program. 

In education, a main criterion for demonstrating validity is 
showing that changes in student performance over time occur as 
the result of college. In contrast, the validity of the end 
product alone rather than how it developed, can be important in 
noneducational settings. In the work world, employers may only 
be interested in selection or retention of employees or in the 
extent to which a candidate for promotion can demonstrate an 
ability, not how or whether the ability was acquired at the 
organization or whether the ability can or should be taught. 
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The way in which a person acquires an ability is critical for 
aducational programs. How persons learn, and how they develop 
outcomes is important information for enhancing the quality and 
effectiveness Of programs. What causes change? If college can 
be said to facilitate change in student performance, then the 
learning proce&s can be said to be valid. 



In 1976, Alverno faculty made a commitment to establish the 
validity of outcomes. They identified several major questions as 
their initial thrust, and we designed an eight year plan for 
carrying out the research objectives operational ized from these 
questions (Mentkowski, 1977b). 

To carry out these research questions, the faculty first 
created a context for validation by establishing evaluation as a 
concept and function, and created an Olfice of Research and 
Evaluation. Evaluation/validation is thus a part of the learning 
process (Figure 2). Estahlishing evaluation/validation as a 
curricular component led to the identification of the following 
five research questions. They are: 



• Are the competences and assessment 

techniques of the learning process valid? 

t How do students change on college outcomes 
described by their potential for cognitive 
development, learning styles, and generic 
abilities? 



• Are outcom^ mirrored in students' 
perceptions of their learning and abilities? 

• How do outcoi^es learned in college relate 
to lifelong learning, abilities, careering 
and professional development after college? 

• What competences describe the performance and 
perceptions of outstanding professionals? 



Each of these questions was operat localized via an overall 
validation design, complete with specitic questions, designs, 
instruments, and methods so a more systematic validPti^^n of 
outcomes could occur. During the past seven years, from 
1976-1983, these questions have been researched with support from 
Alverno College and from a three year grant from the National 
Institute of Education. 



Establishing EvaluationA/alidation 
as a Component of the Learning Process 
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LEARNING PROCESS 



What are the developmental, 
holistic and generic abilities each 
student must demonstrate in order 
that we consider her a !ifelong 
learner? 



How can we develop these abilities 
in each student so they become 
internalized, integrated and 
generalizable? 



How will we know if each student 
has achieved these abilities according 
to our prescribed standards? 



Is the learning process we use to 
develop and assess for abilities 
actually working tne way we have 
designed it? 

Are changes in performance of 
student outcomes related to college 
instruction? What is the relationship 
between current outcomes and 
future outcomes? 

How do current and future student 
outcomes compare against internal 
and external standards? 



What are the outcomes of the 
learning process, those credentialed 
and those expected but not 
credentialed? 



What are alumnae realizations in 
perception, careering and profes- 
sional pe'^fo^.ia^ce as a Ufelong 
learner? 



} 
1 



COMPEl 


FENCES 






EXPERIENTIAL 
LEARNING 






ASSES 
PRO( 
TECH^ 


SMENT 

:ess, 

IIQUES 



EVALUATiONA/ALlDATION PROCESS 



STUDENT 
CHANGES IN OUTCOMES 



} 



ALUMNAE 
FUTURE OUTCOMES 



Figure 2. A description of Alverno program components with evaluatton/vaiidation process. 
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A main outcome of the research is the overall approach to 
validating outcomes that emerged from researching the five 
objectives. It is appropriate here to describe this approach, 
the features of our at*-empt to validate outcomes, and the overall 
validation design to provide the context for the ten r'esearch 
reports that follow this overview and summary. 

Identifying Assumptions About Validity 

Ih^ring our ongoing dissemination of the J.aflu£fl and_ early 

results described in this report, many of our colleagues in 
higher education were interested in a broad overview of how we 
conceptualized validating a liberal arts, outcome-centered 
curriculum as a first step in thinking about the validity of 
their own programs. In order to define "validity" as a concept 
and create a framework for establishing validity of abilities 
learned in college, and to communicate this to our colleagues, we 
set forth our assumptions about validity that were identified, as 
we researched the five questions stated above. 



Validation Is Developmental 

When we create programs, we assume that the program will 
continue to develop. We recognize that most educational programs 
are undergoing various changes, and that new programs have 
start-up time and may then undergo periods ot maintenance. But 
if a program is dynamic and responsive to students, further 
change will continuously occur. 



Where a program is in its development is critical to the 
types of strategies used to demonstrate its valid ^y. "he kinds 
of internal and external criteria or standards to which a program 
is held depends on the extent to which faculty have defined 
outcomes and are able to assess them, the availability of 
information from which standards can be drawn, and also on hoM 
long the program has been in operation. It is hardly conceivable 
to fault a • "Ogram for not having related student outcomes to 
future outcomes if the program is new and does not yet have 
alumnae with extensive post-college exx)erience. If faculty 
define competences or abilities C rather Vxan grade point average 
or subject area tests) as outcomes, and few theoretical 
frameworks for understanding these competences exist , one cannot 
fault them for not establishing construct validity. If there are 
no tested measures of college outcomes avai?able, one cannot 
fault them for selecting new and untried measures as external 
criterion measures. 



Thus, the kinds of validation questions and issues that can 
be addressed by a faculty concerned with validating outcomes is 
limited to a degree by how far the faculty has come in 
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conceptualizing and impleaenting the cuirictilum, and by what 
measures are available for coofparison. This is an especially 
important consideration in validating per£ormance-ba»ed liberal 
arts curricula since they are geneially of recent vintage. 
Indeed, our own attempt .to begin validation research coincided 
with the anticipated gtaduation of our first students fron our 
per formarice-based curriculum^ three yearV* after its 
implementation* 

Validation Is an Jlluminative, Diagnostic Process 

Establishing the validity of college outcomes is never 
"finished." Since^ programs chaag^ and continue to develop one 
cannot and should not consider a program ever completely 
validated. Further, validation strategies are applied »to a 
roiuplex system. Each aspect or level in the system is 
iterrelated with another aspect and level, and every change 
changes evl^iything. As validators, we face a considerable 
challenge in trying to weigh the effectiveness of such integrated 
environments and their element*. Couple this with an increased 
emphasis on "standards rather than normative comparisons, and it 
is cleat we face an enormous complexity in validating outcomes. 
How we approach this complexity — our "mindset" — will impact our 
ability influence the future evolution of higher tJucation 

(Mentkowski , 1980). Because of the complexity of context of most 
programs in higher education and the complexity of abili*:ies and 
outcomes^ toward which one is teac hing, validation efforts cannot 
"prove" val-.dity, , but can illuminate the sjuality and 
effectiveness of programs and the extent to which changes in 
student outcomes are relaL«M to future outcomes, 

# 

To justify the amount of time, effort and resources required 
for validation research, results must be diagnostic. Validation 
results must be usable to improv programs and to continually add 
to the insight faculty bring to teaching, learning and assessment 
issues. Establ ishi. ^ validity means to continue, throughout the 
life of the program, to engage in efforts to bring one closer and 
closer to realizing program goal 3 aud objectives, which also 
chanp.e . 

Validation Relates Theoiy to Practice 
and Research to Evaluation 



VRien liawrenc^ Kohlberg initiated the Just Community approach 
to schooling, he made the leap from theory to practice. This 
step allowed a test of concepts emerging from his theory and 
research studies, and contributed to their credibility for the 
educational world. Some years after this le p to practice, 
Kohlberg confessed to the ."psychologist's fallacy" (Kohlberg, 
1979, in Hersh>, Paolitto & Riemer, 1979) of assuming that 
djevelopmen' 1 theory as exemplified by stages of development 
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could or should form the most* important cornerstone of 
educational practice. This theorist's fallacy has its 
counterpart in the "researcher's fallacy," in which we are 
tempted to assume that the goals, methodology and instrumentation 
that are characteristic of researcji studies seeking theory 
development and demonstrating cau5e-ef f^»ct relationships should 
form the cornerstone of an approach to the practice of evaluation 
and validation. 



While many program evaluation studies in current literature 
se^m to depend almost entirely on the techniques of the 
r^earcher, evaluation has begun to emerge as a separate 
discipline . Evaluators have evolved strategies that clearly 
recognize differences between the purpose of research studies and 
those of evaluation, and have created alternate approaches (Bryks 
1983; Parlett & Hamilton, 1976). This development, as well as 
the growing '-ecognitic : that practitioners are equal partners in 
creating theory ai actice (Kosher, 1977), sets the stage for 
avoiding the "researc 's fw^ilacy." 
« 

A mindset for program evaluation thus begins with the 
awareness that evaluation goals and strategies are better 
selected and derived from ^ the^ practitioner th* a from i:he 
theorist. The question is not "What is available that we can use 
to validate?" Rather, "How might we best analyze the special 
characteristics of this curricultxm so that our validation 
objectives match the nature of the specific program? What is the 
relationship between tools for assessing broad outcomes of 
college and instruments that assess the defined abilities from a 
program?" In the previous section we have described Alverno's 
currlcular goals and theory of ssessment so that a rationale for 
selecting the frameworks and instruments we used to validate 
outcomes could be critiqued. One projected result of this move 
from theory-to-practice and from research-to-evaluation is that 
WG seek to investigate questions suggested by practitioners, and 
to consider the context in %*hich validation is attempted.- 



Validation Is Contextual 



Earlier , we commented on the importance of recognizing 
validation as d developmental process that walks hand in hand 
with the prograiL its methods are applied to. Cltarly, the 
context in which validatlot research is jonducted has several 
important implications for- validation designs and strategies. 



First, we conceptualize validation in aa ongoing, changing 
curriculum where the object of study does not "hold still." 
Second, validation goals and objectives need to be derived from 
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through the validation process. The philosophy underlying the 
curriculum, beliefs about how students learn, and student and 
faculty roles impact the kinds of validation objectives ard 
strategies that can be employed. This need not be taken as a 
negative constraint. Rather, if we are to avoid the asearcher's 
fallacy, then "validity" of validation strategies as that we 
design validation goals and strategies within the context of a 
particular setting. The press of the setting can often serve as 
a guidepost and beacon in validating nontraditional outcomes. We 
benefit from such an approach later when results from validity 
studies are ready to be discussed, critiqued, and ultimately 
implemented . 

Third, the design for validating outcomes needs to flow from 
the structure characteristic of the context. Validating outcomes 
cannot be successfully initiated if the way in which outcomes are 
defined is not considered. Involving faculty and students in 
validation strategies cannot occur unless expectations set for 
their involvement are apparent in the program itself. For 
example, students who come to understand the need for multiple 
assessment of their abilities are more likely to undevstand why 
they aro *>sked to perform on other than faculty designed measures 
(Mentkowski, 1979). Again, rather than being perceived as a 
constraint, the context should be seen as the source for design 
and implementation guidelines. The "validity" check of the 
context is an important indicator of the extent to which the 
results from validation studies are those that are both true and 
useful. 

Defining Validity 

Establishing the validity of programs is a relatively new 
concept. Sets of standards (Rossi, 1982) for conducting program 
evaluations have been formulated, and these standards contain 
601P3 operational advice. The evaluation research community has 
edited a number of voltimes to aid colleagues. But this thrust 
has been a recent development. 

One source of definitions of validity is set forth by the 
measurement community for instrument validation. These types of 
validity have become one way in which the field of educational 
measurement can identify measurement techniques that will yield 
valid, relia)>le scores from which valid inferences can be drawn. 
These standards define validity as establishing content validity, 
face validity, construct validity criterion-related validity, 
predictive validity and discriminant validity (American 
Psychological Association, 1974). 

Since the advent of performance-based education, with its 
emphasis on criterion-referenced measurement, organizing 
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Since the advent of performance-based education, with it8 
emphasis on criterion-referenced measurement, organizing 
validation efforts around these types of validity has proved 
difficult (Grant, Note 2; Messick, 1980; Pophara, 1978). When 
first faced with these issues in 1976, we attempted to simply 
modify the existing types of validity (content, face, construct, 
criterion-related, predictive) to fit the outcomes and assessment 
techniques we were validating. That attempt failed. The 
purposes and characteristics of instruments have changed, and we 
now need assessment techniques designed to measure abilities 
which consider the role of assessment techniques and processes in 
the teaching/ learning process, the need for demonstrating the 
equity of the instrument and the importance of giving feedback to 
students. Governmce questions related to who decides on 
criteria and standards are also an issue. Often, we do not have 
a clear picture of the complex constructs we are trying to 
measure. They are often developmental constructs, and we expect 
change. Test/retest reliability is therefore not a goal. Nor do 
we expect that abilities developed in college will have a 
straight 1 ine prediction to how they are demonstrated after 
college or even how they are defined. We are interested in 
developing abilities. Prediction to success college is not as 
important as having diagnostic information on which to build 
instructional practice. Other issues relate to effective 
approaches for establishing the validity of programs, assessment 
techniques and outcomes which focus on the need for evaluation as 
well as validation efforts, and which consider the contextual, 
developmental and illuminative nature of programs (Weiss, 198'^). 
We soon came to realize that we needed to rethink validity based 
upon our new assumptions about its use and function. Faculty 
questions provided the fri>^'»work f^r designing a validation model 
and creating validaf.ion strategies. The nature of the questions 
and their relationship to various aspects of the learning process 
model (competences, experiential learning, assessment process and 
techniques) will be discussed in the next section. 

Out of this experience, we have come to think of two types of 
validity, design-based validity and per formance-based validity. 
With design-based evaluation and validation strategies in place, 
the research results from per formancp.-based validation strategies 
are more likely to be incorporated into program development 
efforts. If a program is constantly changing and assessment 
techniq'ies , )nsistently revised, new information has a place to 
go — a place lo begin to be tested in the practical context from 
which it arose. For each of the two types of validity, we later 
specify the nature of the questions asked by* faculty, which 
determine comparisons against internal criteria or standards and 
those external to the program, and how these compai'isons will be 
effected . 
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Design-Based Validity 



Design-based validity has its basis in criteria which faculty 
use to define competences, develop learning strategies, and 
design an assessment process (Figure 1). But as every curriculum 
designer knows, what looks good on paper needs adjustment and 
monitoring to make it work in actual practice. Program review 
and monitoring procedures are critical to establishing 
design-based validity. But how does one know a program is 
meeting these goals? Design-based validity refers to those 
strategies that monitor program function and compare the program 
against criteria or standards evoked during program design ("What 
is occurring? How is it occurring? What should be occurring?"). 
Both internal and external criteria or standards about how 
program components should be designed are used to answer the 
q^^stion "What should be occurring?" both during design and 
implementation. For example, identifying competences , learning 
strategies and an assessment process evolved from the expel"t 
judgment of faculty (internal criteria or atandards) who had vast 
experience teaching and assessing students. Faculty drew on this 
experience to create the various components of the learning 
process . For example, one design criterion or standard for 
defining competence is that competences be defined as 
developmental, holistic and generic. A criterion or standard for 
developing assessment techniques is that the performance mode be 
similar to the ability as it is usually expressed. The corporate 
faculty pooled their resources as designers. Thus, one source of 
criteria or standards is the expert judguent of the faculty. 

Design-based validity does not necessarily rely entirely on 
faculty judgment based on their own criteria or standards. 
Criteria or standards from outside the college (external 
standards) are drawn from various sources. For example, 
professional groups were consulted on the definition of 
abilities. Expectations about the nature of the abilities needed 
r'n personal and professional role* of graduates (future outcomes) 
t^ere ulso discussed. Literature reviews were also used. 

A program can be said to have design-based validity when the 
comparison between what is intended and what is actually 
happening on a day-to-day basis at any one point in time is 
realized. Thio comparison is effected through a variety of 
^?'^^^*LP^P^-^4pres carrieil out io^xelation to various- aspect* of 
the curriculum (e.g., Assessicent Coinnittee evaluates instruments; 
syllibi are submitted for review; external assessors from the 
Milwaukee community judge student performance and critique the 
assessment process) (Mentkowski, 1980). 
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Performance-Based Validity 



Design-based validity alone can be tautological. Even though 
designers and implementors consult outside resources, there is a 
need to measure program outcomes. In our case this means the 
performance of students. Performance-based validity refers to 
the strategy of reviewing student performance of outcomes as it 
develops through instruction rather than comparing how the 
program functions against internal and external criteria or 
standards ("What is occurring, how is it occurring, why is it 
occurring?"). Validity rests on whether student performance 
changes over time as the result of instruction, and whether these 
changes persist beyond college. At the sair time, one cannnt 
stop there . "Is the change in student per fonnance ' good ' 
compared to a standard?^' is still an important question. 



Thus, changes in student performance need to be compared 
against criteria. For example, suppose that a study of student 
performance shows change on a faculty designed m:asure of 
analysis . The faculty can ask, "How does the range of 
performance compare with how we have defined the ability 
(internal criterion or standard)?" They may also ask "How do 
students perform on an external criterion measure of analysis 
developed by this researcher of analytical thinking?" or "How 
would professionals in management demonstrate analysis, and do 
our students show the beginnings of this ability as expressed 
on-the-job?" (external criteria or standards), 

* 

One of the first questions we must deal with in measuring 
outcomes _-and-ljataxre^outcximefl-J.s , tJxe., identificati^ 
criteria to which outcomes will be compared. 'The basis for 
establishing validity is comparison. But what should comprise 
the nature of tne comparison? Whose standards, and whac kind of 
standards are adequate? The search for standards *-0 which 
program and student outcomes can be compared is a continuing one. 
In the section "Defining, Assessing and Validating Outcomes in 
Higher Education," we pointed to the lack of frameworks and 
measures available for use as standards to which a college's 
outcomes could be compared. We have chosen those frameworks and 
measures more likely to meet certain of our own criteria for 
outcomes and measurement techniques. Yet, we recognize that 
choice or selection of any criteria — whether a measure of 

—cognitive devel o p m e nt , - a- s e t — of- abilit "ie» t h a t — d o sc ribe- 

professional perfomance at work, a set of norms based on a range 
of studeat performance, advice from a group of external 
assessors, or goals from program designers—is somewhat 
arbitrary. We deal with this question by using a variety of 
approaches to establishing validity, using a variety of criteria 
or standards from sources both internal and external to the 
learning process, and the performance of our students. 



However, faculty selecting a standard should consider its 
representativeness or the extent to which the standard is 
inclusive of the interest group. Second, a standard should be 
valid. To what extent is the external standard meaningful? If 
both these questions cannot be answered to one's satisfaction, 
the external standard itself may need to be validated before 
including it in a validation study. Thus, types of criteria or 
standards include those developed by faculty as well as those 
identified outside the institution. 

In addition to focusing on questions about changes in student 
performance over time, performance-based validation strategies 
e ^mine the relationship between the program and student 
performance. This comparison allows us to determine the 

effectiveness-^ f -the curricuLuni^-- or rather^ the — inter ac^ions^- 

between the curriculum and change in student performance. This 
comparison is effected by observing changes in student 
performance over time in relation to educational experiences 
(instruction). The results of design-based validation studies 
are thus further challenged by performance-based validation 
strategies, just as the results of studies of changes in student 
performance are further challenged by comparison to external 
standards . 

T'us, the true test of a program (design-based validity) is 
its relationship to changes in student performance over time 
(performance-based validity). And the true test of student 
performance is to examine how student performance changes ever 
t ime in rel a t ion t o edu^^ ^ expexienc^e^ ...wlieth^|^_iJl5Pe 

changes persist beyond college. Ultimately, both the degree and 
type of change in student p .ormance of outcomes over time is 
compared to internal and external standards. 

Making the shift from the traditional types of validity to 
design-based and performance-based validity helps to 
conceptualize validity given the assumptions that have been 
previously specified about its role and function. Validity is 
developmental, a process, considers theory-practice, 

research-evaluation relatedness, and is contextual. Strategies 
for establishing performance-based validity are ongoing. In a 

co ntinugt'flly ch anging p^^^gr^i desigxt-jjased --vali44-ty— ^al^- 

ongoing. Redesign is often concurrent with attempts to establish 
performance-based validity. We cannot expect that a faculty 
carry out performance-based validation strategies on alumnae 
until there are graduates, nor can a faculty validate criteria 
for assessment until outcomes have been identified and defined. 
In general, however, attempts at performance-based validity will 
be imultaneously attempted with design-based validity 
strategies . 
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In some ways this is an advantage. As stated earlier, with 
designed-based evaluation and validation strategies in place, the 
research reaults from performance-based validation strategies are 
more likely to be incorporated into program development efforts. 
If a program is constantly changing and assessment techniques 
consistently revised, new information has a place to go^^a j)lace 
to begirr to be tested in the~pr act icaT^olirext from it 
arose . 



Identifying Validation Questions 

As stated previously, our assumptions about validation and 
our definition of validity arose from questions faculty began to 
ask as they designed, implenented and tested the curriculum 
again st — sirtident — petceptiuus — and' per formatter, ^^tn an ongoing 
curriculum these questions continue to be asked, since it is 
unlikely that a changing curriculum will ever be validated in an 
absolute sense, nor do we think it should be. 



Earlier, we categorized questions into four general kinds: 

• Descriptive questions; "What is occurring?" 
"How is it occurring?" 

• Ascriptive questions: "Why is it occurring?" 

• Evaluative questions: "Is v^at is occurring 
'good* compared to a criterion or standard?" 

-t h e s t-andOTd- vfftitl?^ ^ 

• Presc iptive questions: "What should be 
occurring?" 



Establishing design-based and performance-based validity 
means applying these questions simultaneously to the curriculam 
components and to student per formance of current and future 
outcomes. Asking descriptive questions implies observation and 
measurement of eh_ange« in student p^erformance over time. Aslcing 
apcriptive questions implies establishing relationships between 
various curriculum components and current and future outcomes . 
A^ing ^valua^ive quesTT^ons implies a comparison betw^?en 
curxiculum components and student performance of outcomes, to 
internal and external standards, and asking if those standards 
are valid. Asking prescriptive questions implies impl ^-ment t ng 
research findings to improve current andet standing of st udent 
needs and curriculum practice. Because the questions are applied 
in an ongoing and changing curriculum, there is a need for 
invest ip^t ing all four types of quest ions simultaneously . In 
order to respond to these quest ions, faculty created an 
additional component of the curriculum in addition to 
competences, experiential learning and assessment process. This 



component is c«iled evaluation/validation process and techniques 
(Figure 2). In order to establish design-based validity, the 
faculty created internal review, evaluation and revision 
mechanisms at the same time as the program was designed. 

An Office of Research and Evaluation was created three years 
after program implementation to establish performance-based 
validity. Faculty questions that stimulated the more systematic 
performance-based validation research through the Office of 
Research and Evaluation can be categorized with reference to the 
curriculum component against which it is applied (competences, 
experiential learning, assessment process , student outcomes, 



• Are our assumptions about the complex 
nature of each competence adequate? How 
best should the ability be defined so that 
its meaning is clear? Have all aspects of 
the ability been defined? 

• Are the competence ^vels actually sequential? 
Is one competence level necessary in order to 
demonstrate the next level? 

• Is each competence level more complex than 
the previous one? Does the next level 
appear more complex only because it is 
integrated with1?ore complex content? 

• Have all the significant relationships 
between the competences been identified? 

• Ar^— ast>ects of ^n— ability oi^competpnce 
common or generic to each discipline 
identified and measured? 



Validation of Competences Compared 
to External Criteria or Standards 

• What competences do professionals perceive 

.IS critical for outstanding performance, 
— e<i^tteatien itnd selection? — 





Faculty Questions for Establishing Validity 



Validation of Competence Compared 
to Internal Criteria or Standards 
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• What competences do effective professionals 
perform? 

• How do professionals describe their careering 
and professional development? 

Validation of Experiential 
Learning Compared to Internal 
Criteria ojr Standards 

• Do learning experiences reflect the basic 
tenets of experiential learning in both 
classroom and field experiences? 



VaHdation of Ex perien tial 

Learning Compared to External 
Criteria or Standards 

• What gains or cnanges in performance do 
students demonstrate as a result of the 
learning process? 

• To what aspects of the learning process do 
students attribute theii development? 

Validation of che Assea^MI"^ 

Process Compared to Internal ^ 

Criteria or Standards 

• Are criteria used to judge performance 
in relation to the competences valid? 

• Is the instrument stimulus and mode of 
assessment appropriate? 

• Are the judgments of performance reliable? 

• Do assessment techniques measure the effects 
of instruction? 



Validation of the Assessment 
Process Compared to External 
Criteria or Standards 

# How does the assessment process compare 
to assessment center standards? 
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Vulidation of Changes in Student 
Outcomes Compared to Internal 
Criteria or Standards 

• What is learning to learn? 

• How do students learn to learn? 

• Hov do students learn to learn from 
experience? 

• How do students learn to learn on the job? 

— ^ What are the 1 earning" outcomes or^rbcesses 
each student is able to demonstrate? Are 
outC<5m6S defined ill" "ways that reflect what 
we understand about students and the 
development of the abilities? 

• How do abilities or competences develop? 

• To what extent are abilities or competences 
developmental? Are they teachable? 

• To what extent are abilities or competences 
holistic? Are they internalized and 
characteristic of the person? 

^ To^-i^t -extent— are- abi^rirtie^- or co m petences 

generic? Do students generalize their 
performance across time and situations? 

Validation of Change in Student 
Outcomes Compared to External 
Criteria or Standards 

• How do students change on college outcomes 
described by their potential — what is 
possible for them to achieve? 

• How do sTudent outcomes compare with 
outcomes from tft^dents at other colleges? 

• How are outcomes we assess for mirrored in 
students' perceptions of their developing 
abilities? 

• How are outcomes, abilities or competences 
achieved in college causally related to 
effective performance of professionals 

at vork? 
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Validation of Future Outcomes 
Compared to Internal Criteria 
or Standards 



• What are the future outcooiesy abilit\es 
or competences alumnae demonstrate in 
their professional performance? 



• How are alumnae outcomes ve identify 
mirrored in their perceptions of their 
developing abilities? 

• How do alumnae transfer abilities to life 

^fteT~c6llmgef~~^-~^ — _ - - 

• How is Hfelong learning characterized? 

• How do alumnae demonstrate careering and 
professional development? 

• How do alumnae relate personal and 
professional rOles? 



Validation of Future Outcomes 
Compared to External Criteria 
or Standards 

• How are outcomes learned in college 
related to graduates' future personal 
apd professional performance? 

# How do alumnae future outcomes compare 
^o those demonstrated by outstanding 
professionals? 
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RESEARCH OBJECTIVES AND REPORTS 

Questions faculty asked about the validity of college 
outcomes were operationalized into research objectives within an 
overall validation model. The model included specific questions, 
designs, Instruments, and procedures so the more systematic 
validation of outcomes might occur. While we argued that faculty 
questions are researched simultaneously to a degree, we also 
realize that validation is developmental, and that it will be 
carried out within a particular context. We are, therefore, 
selective in carrying out a program of research which may be 
directed to most, but not all, components of the validation u<odel 
at a particular time. 



The assumptions and faculty questions presented so far in 
this report contribute directly to the research objectives 
specified in the prior reports to the National Institute of 
Education for the grant "Careering After College: Establishing 
the Validity of Abilities Learned in College for Later Success" 
(Mentkowski & Doherty, 1977, 1979, 1980a, 1980b). These research 
objectives are: 

To internally validate the competences and assessment 
techniques of the Alverno learning process by — 

I. Seeking to establish the validity of the 

tecliniqu^ff usied iro ass^w student 

performance by adapting or developing 
validation techniques appropriate for use 
with nontraditional assessment instruments; 

II. (a) Comparing student performance across 

and within competences to further refine 
the nature of the competences and their 
interrelationships ; 

(b) Examining the relationships between student 
performance and external criterion measures. 

To externally validate the student outcomes of tlie Alverno 
Collage experience by — 

III. (a) Comparing the competences identified by 

Alverno with the competences demonstrated 
by outstanding professionals; 

(b) Following the future careering of our 
graduates in their various professions 
after college; 



^3 5:j 



IV. (a) Generating in-depth profiles of student 
perceptions of themselves and their 
development and analyzing the relat^n- 
ship of these perceptions to Alverno's 
learning process; 

(b) Assessing student attitudes toward the 
learning process; 

y* (a) Assessing students on cognitive-development a'' 
outcomes identified as descriptive of 
individuals who have reached various levels 
of potential in ego, moral, and intellectual 
development; 

(b)_JLaaeasing students oa ^em?r ic-c omp e t e nc e 
externr"* criterion, measures that assess a 
variety of analytic and interpersonal 
abilities. 



The ten research reports that comprise the full report 
respond to the objectives as initially stated. The more specific 
questions that followed fiom these objectives have been stated 
earlier. They are formulated to best communicate results to the 
more general higher education audience. 



Therefore, the five questions listed below structure che 
fcomplete report* 

• Are the competences and assessment techniques 
of the learning process valid ? 
(Objectives I and II above) 

• How do students change on college outcomes described 
by their potential for cognitive develojpment" j 
learning stales and generic abilities ? 
(objective V above) 

• Are outcomes mirrored ^*n students' perceptions 
of their learning and abilities? 
(objective IV abave) 

• How do outcomes learned in college relate 

to lifelong learning, abilities, careering and 
professional development after college ? 
(Objective III, b) 

• What competences describe the performance a nd 
perceptions of outstanding professionals? 
(Objective III, a) 
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Each of these five questions is related to one or several of 
the faculty questions listed previously. At the sme time, each 
represents a separate, involved researcit approach. The 
relationships between each o^ these approaches are apparent from 
the previous section. We will later provide links bttween the 
conclusions we draw from each of the five research th^us^s, and 
describe implications for validating the outcomes of 
outcome-centered }.iberal arts curricula. 

So that the reader may relate thest questions to the 

components of the validation model presented next (Figure 3), 

each of these five questions is listed again, with the ques^ions 
at form the basis for each study. 

• Are the coapet nces and assessment techniques 
of ^ he learning process valid ? 

• Do competences reflect our understanding 
of how they develop? Are competences 
developmental? 

• Do compet<=*nces involve the '*ole person? 
Are competences holistic? 

• Do competences general^'ze across time 
and situations? Are compet^ces generic? 

• Are assessment criteria valid? 

• Is assessor expert j\idgment reliable? 

• Do instruments meabure the effects 
of instruction? 

Friedman, M., Mentkowski, M., Earley, M., Loacker, G., & 
Diez, M. Validating assessment techniques in an 
' outcome^centered liberal arts curricu^ ^!m: Valuing 

and comfiunications generic instrument . Fiaal Report to 
the National institute of Mucation, Research Report 
Number One. Milwaukee, WIf Alverno Productions, 1980. 

Friedman, M., Mentkowski, M.,'Deutsch, B., Shovar, M. N., & 
Allen, Z» Validating assessment techniques in an 
outcome-cenLered liberal arts curriculum: Social 
interaction generic instrument . Final Report to the 
National Institute of Education, Research Report 
Number Two. M'iwaukee, WI: Alverno Productions, 1982. 
\ 

Mentkowski, M. , O'Brien, K., Cleve, L., & Watzdorff, A. 
Assessing experiential learning: The learning incident 
as an assessment tech- ' ique . Milwaukee, WI: Alverno 
Troducticns, 1983. 
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• \hat evaluation^ revision and validation 
techniques ar ~ more appropriate fpr 
nontraditional assessment techniques? 



• Which generic sstssments a^e better 
indicators of college performance and 
performance characteristics thaf: can serve 
as cross--discipl inary ou tcome measures? 

• How uo students change on generic measures 
of student performance? 

Alverno College Assessment Committee/Office of Research and 
Evaluation. Validating assessmen^ techniques in an 
outcome'"centered liberal arts curriculum: Insights from 
the evaluation and revision process" ! Final Report to 
the National Institute of Education, Research Report 
Number Three. Milwaukee, WI: Alverno Productions, 1980. 

Alverno College Assessment Committee/Office of Research end 
Evaluation. Validating assessment techniques in an 
ou t c ome- cent er e d l iberal arts curriculum; Integra ted 
Competence Seminar . Final Report to the National 
Institute ot Education, Research Report Number Four. 
Milwaukee, WI: Alverno Productions, 1982. 

Alverno College Assessment Committee/Office of Research and 
Evaluation. Validating assessment technioues in an 
outcome-centered liberal a rts curriculum: Six Performance 
Characteristics Rating . Final Report to the National 
Instituts of Education, Resef h Report Nuiu.ar Five. 
Milwaukee*, WI: Alverno Productions, 1983. 

Alverno College Assessment Committee/Office of Research and 
Evaluation. Six Performance Characteristics Ra ting. 
Milwaukee, wfi Alverno Productions, 1978, Revised 1979. 



Question II • How do students change on college outcomes described 
by their potential for cognitive deTelopme nt, 
learning styles and generic abilities ? 

\^ 

• How do students cbange over time on 

measures of '-nman potential—cognxtTve 

development, leari^ing styles and generic 
abilities? 

• (5kn change be attributed to performance in 
the learning process rather" than to 
differences in age, background, or college 
program? 
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• What patterns of change emerge in the 
interrelationships of the human potential 
measures of cognitive development, learning 
styles and generic abilities, and generic 
measures of college performance? 

i 

I Me*-itkowski, M. , & Strait, M. A longitudinal study of 
student change in cognitive development, learning 
styles, and generic abilities in an outcomes-centered 
liberal arts curriculum . Final Report to the National 
Institute of Education, Research Report Number Six. 
Milwaukee, WI: Alverno Productions, 1983 • 

Nentkowski M., Moe ser, M. , & Strait, M. Using the Perry 
scheme intellectual and ethical development as a 
coileg ^ tcomes meajure: A process and criteria for 

j udging student performance . Vols. 1 & 2. Milwaukee, Wl; 

Iverno Productions, 19C3. 



Mentkowski^ M., MiJJer,. N.^ Davies, E.* Monroe^ E.^_& 
Popovic, Z. Using the Senten^p Completion Test 
measuring Loevinger's stages of ego development as a 
college outcomes measure: Rating large numbers of 
protocols and maintaining validity of the ratings . 
Milwaukee, WI: Alverno Productions, 1981. 

Mentkowski, M. Changes in student profiles on the Learning 
Style Inventory . First Report to Participants in a 
Longitudinal Study of College Outcomes. Milwaukee, WI: 
Alverno Productions, 1981. 

Mentkowski, M. Understanding the development of thinking 
in college . Second Report to Participants in a 
Longitudinal Study of College Outcomes. Milwaukee, WI: 
Alverno Productions, 1981. 

Mentkowski, M. Some questiotio and answers about evaluation 
studies . Third Report to Participants in a Longitudinal 
Study of College Outcomes. Milwaukee, WI: Alverno 
Productions, 1979. 

Mentkowski, M., & Fowler, D. Learning to learn ac work : 
Students, alumnae and other professionals ^ Fourth 

- — ilepart- to P^rt^i^ ipant s in^ a-4,oBgit«diiw»l St«dy of 
College Outc ome s . Mi Iwaukee , WI : Al srer no 
Productions, 1981. 



Question III » Are outcomes mirrored in students* perception s 
of tiieir learning and abilities ? 

• How do students understand and justify 
learning outcomes? 
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• How do students understand outcome-centered 
liberal learning as relevant to performance in 
personal and professional roles? 

Much, N., & Mentkowski, M, Student perspectives on liberal 
learning at Alvernn College: Justifying learning as 
Irelevant to performance in personal and professiona l 
roles . Final Report to the National Institute of 
Education, Research Report Number Seven. 
Milwaukee, WI: Alverno Productions, 1982. 

Mentkowski, M. Alverno College Attitude Survey . 
Milwaukee, WI: Alverno Productions, 1977. 

Mentkowski, M., & Much, N. Alverno College Student 
Perspectives Interview . Milwaukee, WI: Alverno 
^ Productions, 1980. 

Mentkowski, M., & Bishop, J. Alve rno Col lege Student 
-Career ing Que^tionn a ire , ^rlwenkee, WI: Alverno 
Productions, 1981. 

Question IV • How do outcomes learned in college relate to 
lifelong learning, abilities, careering a nd 
pro f ess ional ^ ^v's iQp^^p ^ ^ ^ j 

• How do expectations of students and 
realizations of alumnae compare? 

• What abilities and processes^ ^nable 

. trangfpr nf Iparni no tO prof PSsionfll _ - 

performance and careering after college? 

• How are alumnae learning to learn at work, 
and do they describe lifelong learning? 

• What are alumnae perspectives on careering 
and professional development? 

Mentkowski, M., Much, N., & Giencke-Hol 1 , L. Careering 
afte r college: Perspectives on lifelong learning and 
career development . Final Report to the National 

^" Institute of Education, Research Reporr l^ber Eight . . — 
Milwaukee, WI: Alverno Productions, 1983. 

Mentkowski, M., Much, N., & Giencke-Holl , L. Careering 
after college . Fifth Report to Participants in a 
Longitudinal Study of College Outcomes. Milwaukee, WI; 
Alverno Productions, 1984. 

Mentkowski, M., & Much, N. Alverno College Alumna 
Perspectives Interview . Milwaukee, WI: Alverno 
Productions, 1980. 



48 5 8 



Mentkowski, h. , & Bishop, J. Al ino College Alum na 
Careering Questionnaire . Milrffukee, WI: Alverno 
Productions, 1980 • 

• What perspectives and strategies do 

alumnae demonstrate in relating personal 
and professional roles? 



Question V • Vjhat competences describe the performance and 
perceptions of outstanding*^rof essionals ? ^ 

• Vhat competences do outstanding 
professionals in nursing and management 

.^^^^^^^^yTJerfOrm? 

• What competences do professionals in 
nursing and management perceive as relevant 
to performance, critical for education and 
selection, and descriptive of outstanding 
performers? 



• How do professionals describe their careering 
and professional development and what aspects 
are related to performance? 

Mentkowski, M. , DeEack, V,, Bishop, J,, Allen, Z,, & 

Blanton, B, Developing a professional competence model 
for nursing education . Final Report to the National 
Institute of Education, Research Report Number Nine. 
Milwaukee, WI: Alverno Productions, 1980. 

MentkowsM,~irrra^rrien~ X.T^ Fowler, D. 

Developing a professional competence model for 
management "liduc at ion . Final Report to the National 
Institute of Education^ Research Report Numbei^ Ten. 
Milwaukee, WI: Alverno Productions, 1982. 

Mentkowski, M. , O'Brien, K. , McEachern, W., & 

Fowler, D. Developing a professional competence 

m odel for management education . Fir.al Report Summary 

Tor Participants. Milwaukee, WI: Alverno Productions, 1983 

Bi$hop, J., Jlentkowski, H. ,^aLErien, K., Birney, R,, 
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The sample consisted of over 75U women students ages 17-55 at 
entrance and over 60 two-year alumnae ages 23 to 52 at Alverno 
College. Over 80 women nurses and over 100 women managers anc* 
executives ages 26*66 from the Milwaukee community con prised the 
sample for the professional studies. 

Characteriftics of the Validation Model 
Correlational Rather Than Experimental Designs 

Program development is multifaceted . Therefore, so is a 
validation design. We use multiple approaches^ and demonstrate 
validity through establishing relatedness , rather than by 
establishing cause and effect relationships. Because the 
outcomes are developmental and the curriculum is changing, we 
must use correlat^ional rather than experiniental designs. Xf o^e 
cannot design laboratory studies that will establish cause and 
effect relationships then one must capitalize on correlational 
relationships, and that demands a model where the questions asked 
are in relationship to each other. The findings from one set of 
questions have implications for another. 

For now, we have abandoned most experimental designs and 
methods for establshing validity". The emphasis is on cbmparison 
^ f " ch a n g e s Tn student p erTo rm a n ce ove r~time aga ins t internal and 
external standards. We are not like^lx_tp use grk^uji^gBp^xi^oxi^^ 
designs where one group consists of Alverno students, and another 
consists of students at c college which attracts students of 
similar demographics but does not have a performance-based 
curriculum. We have found that we cannot make accurate enough 
assumptions about where Alverno students and those fi'om another 
college would be similar or different. Thus, the adequacy of 
such comparisons for providing accurate and useful results is 
highly questionable. And selecting a control college is 
impractical. We cannot really "prove" whether a constantly 
ch^mung-jftnd evolving, aurriculum^ i«-ef f active or ineifwtiv^ — b^y- 
using such experimental models. By c:xnparing oui* students 
against external standards, however, we may have some indication 
of how our students compare to scudents at those colleges where 
similar instruments are used (e.g., Winter, McClelland & Stewart, 
1981), and a range of student groups contribute to general izable 
"norms." Clearly, all standards of this type arise piVtly from 
normative data. 

In addition, all students complete at least four levels of 
the learning sequence. There are no intra-insti tutional control 



"60 



groups. Faculty who may not explicitly teach students a 
particular ability are aware of it and may still teach it 
implicitly.^ We h«v« 4>eguii t<y mternally compare students Who 
complete four levels of an ability with those who go on to levels 
5 and 6 as part of their major field, but the currently available 
criterion measures, for the most part, measure only small parts 
of the conjplex abilities demonstrated at level 6. Comparing 
alumnae who graduated prior to implementation of the 
performance-based curriculum with more recent alumnae is also 
unwise* The new curriculum h«d too many of its roots in the old; 
particularly in some academic departments. And the effects of 
the women's movement on careering outcomes could hardly be 
separated from effects of the new curriculum. A developmental 
framework cautions us that abilities learned in college may not 
be visible in the same form in later years. The predictive 
validity of an ability may be difficult to establish if we look 
for "more of the same" in a follow-up study of graduates, rather 
than evidence that an ability is developing and achieving 
intag^rat ion with oth e r - abilities. ~^ ~~ ^ 

How AlvertiO students as a "group compare normatively to 
students at other colleges receives less emphasis than how our 
students' individual gains over four years compare to (l) 
developmental norms, and other standards derived from the 
faculty's understanding of the abilities they teach toward, (2) 
students' perceptions of t ,eir own growth, and (3) standards 
drawn from external criterion instruments that most nearly 
approxima^-e the measurement of the abilities that we tt^ch 
toward. - - — - — 



- ^t we have-tvilT swertfl chaFac teri sties Into 
allow us to move beyond some limitations in correlational 
designs. First, we use an aggregate^ triangulated model. 
Second, we use both longitudinal and cross-sectional designs that 
compare age and class cohorts. We use a time series design to 
attribute change to college performance, and match comparison 
groups for persistence in college. These characteristics of the 
validation model and longitudinal design are discussed below. 

— - — ^ AggregaterTrtangutaWd^MocleT ' 

As far as possible, we approach every outcome or factor we 
study from several directions. Creating models which ask 
questions simultaneously and focus on r^latedness resulb in 
circularity of results and require that we use multiple sources 
of standards and study the development of multiple outcomes. We 
employ triangulation , which means that we measure the development 
of multiple outcomes and avail ourselves of multiple 
opportunities to ompare student performance against multiple 
standards. As stated earlier, a changing curriculum does not 
allow lor using experimental designs to rebearch ascriptive 
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questions. ThuS, we rely on aggregate findings; if we are able 
to demonstrate results in an aggregate, ot„vairiety. of . ways, we 
will have more confidence that our observations are true and 
replicable. 

One of the values of using aggregate findings and 
triangulation is that most questions related to the validity of 
programs are being asked somewhere H the model. While research 
takes time and effort, especially longitudinal research, some 
results with respect to a particul<>r question are usually 
available. Since the research is carried out by in-house staff, 
they are aware of most sources of data and what is currently 
known regarding a particular issue. 

Here is an example of approaching an outcome from several 
^rec^ tions, ConBiHe r the compl j3Hr ty of the pr oc ^s-^cuTty^Iii^e 



m mind when they use a phrase like "analytic thinking." 
Clearly, no single measure—whether devised by a faculty member 
for inatrucJiion and aaae^sment or ixy a team of psychometric ians 
for research—can hope to capture the whole of such an activity. 

So we aggregate several measures of different kinds, each of 
which bears upon some portion of the domain "analytic thinking," 
and t^»^es several approaches at once. We can thus develop a 
general sense of whether something in that domain is changing, 
ano^ can begin^^o ask questions abQulLjdiat.it might b^ and why it 
Is changing based j)n the_ diff^^titial reapoMe* the varied 
measures yield. 



Using triangulation helps solve other design problems . As 
stated earlier, experimental aesigns utilizing control groups are 
inappropriate in a changing curriculum. All students ^ experience 
the "treatment" and even f.ast alumnae, students from other 
colleges or persons not in college do not meet the criteria for 
serving as controls. In this ..anner we may pool Successive 
results on in-class . performance assessments from several 
different disciplines, results frorn^ severjil widely avai lable, 
^easurcs^ ^of-ncognittve"~^ev^^ aiidAor ~ analytic reasoning 

(human potential measures), and results on student perceptions 
from the sequence of open-ended interviews. With this approach, 
«^Ai.L ourselves of at least thrae independent sources ttrat are 

ea» -add" to our 

understanding of more specific questions, while recognizing that 
results must be confirmed from other independent sources. We 
see, from several angles, phenomena we know are difficult to 
research given the practical limitations imposed by real-life 
rather than laboratory settings. Using three groups, students, 
£iunmae, and other professionals , allows another set of 
comparisons at a different level of analysis. 
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Validation Model Components 

Figure 3 presents components of the validation model. It 
diagrams the three major approaches to validating outcomes in 
relation to each other ( per formanc e, potential , perceptions ) to 
illustrate relatedness and triangulation • The model also 
utilizes students , alumnae, and other professionals (see page 92). 

The sources of data have been identified and placed in 
relationship to each other in the model (Figure 3) in order to 
better describe the opportunities for relatedness and 
triangulation . It is apparent that if questions are studied 
simultaneously I there are many opportunities for the outcomes 
defined to be further described and elaborated, and the results 
compared, as the studies continue (see pa^? 95 and following). 



Establishing relationships between changes in outcomes during 
college and future outcomes, is a complex task. It seems clear 
to us that the measurement of such complex outcomes, and the 
measurement of change will proceed with many difficulties. There 
is no simple one-on-one match betwean any of the outcomes, nor 
between outcomes and future outcomes. Clearly, the several 
approaches attempted simultaneously in Figure 3, while reflecting 
the complexity of questions asked, also demand a sophistication 
of strategies and instruments that we have not yet achieved in 
higher education. 



Since jour purpose is to djy^ejpj*^^ 
process, we can proceed with our work in bpite of the pressure 
that cc ^ from researching evaluative questions, to demonstrate 
the rthwhileness" of the program and to show gains in 
performance. Indeed, given the state of the art in measuring the 
complex outcomes of college, we can make progress in some cases 
by describing those outcomes initially, and later asking 
questions of evaluation. 

Such a^complex model .caa_._ajaem - averjAtelmixig^ But — we — hav«- 

found it helpful to specify our questions and some ways in which 
we can begin to ask them. Asking the questions, rather than 
putting them off 4intil adequate designs, strategies or 
in st r uments are availabie^, seems to us a "better way^~ to gfappTe ~ 
with-- their e-ow^^lexitiesv- The urodcl is art trpptrrttniity not to ^ 
generate perfect "results," but to enable us to ask better 
questions. 

From the research questions, we have determined the basic 
structure of a model for validating the curriculum and changes in 
student performance. It is clear that our task is to identify 
and measure changes in student potential . We must obtain a 
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Figure 3. Componamts of a validation model for the Alverno learning process. 
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description of chang^^ in sTudent potential over time in relation 
to student achievement of competence in the learning process, to 
establish the relationship between student potential and 
performance in the learning process. In addition, we must 
identify and measure future outcomes of graduates, and identify 
relationships- between st udent and alumna outcoues. We must also 
identify and asses s student perxeptionfl^ since-_^tudent 
perceptions arc external standards against which performance and 
potential can be compared. We must also identify and assess the 
performance and perceptions of alumnae and other professionals. 



Characteristics of Research Designs 

Longitudinal and Cross-Sectional Approaches 

Rather than equate change with average gains, faculty are 
interested in the extent to which each student changes. The 
curriculum may facilitate growth only for students who are 
verbally, skilled. Or students who enter college with already 
sophisticated abilities may coast through a portion of curriculijm 
and make few, if any, gains. For facilitating individual 
patterns of change and growth, faculty designed the curriculum to 
include cons6ci*tive assessments throughout a student's college 
career. Consequently, we are likely to select similar 
longitudinal designs. While longitudinal studies using external 
criterion measures are — time-con^stnning and costly, ttrey yield 
individual growth patterns. The following diagrams provides a 
picture of the combined longi^udiivat and cross-sectional designs 
used in our study of student and alumnae outcomes. Each dotted 
line represents one of three student groups assessed repeatedly 
in a time series. Each dot on the line represents an assessment. 
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HUMAN POTENTIAL MEASURES: LONGITUDINAL DESIGN 
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Total Sampling 

Students drop out of college and new ones enter in midstream. 
To enable longitudinal research with adequate sample sizes, we 
use total sampling involving all students entering or graduating 
in ^ particular year, rather than random sampling. 



Age and Age Cohort 

Prominent in our report is our concern with age and age 
cohort differences. Because our student population ranges in age 
from 17 to 55 years, and because we expect to continue to attract 
older students in the future, we have a special opportunity ) 
examine change across a larger range of adult life. We have u. i 
age, broken down into traditional and older student cohorts, to 
compare the general influence of life experience, or 
"maturation," to formal education experience. We have also used 
age, standing again for life experience in general, as the 
logical first cause of differences in development and other 
abilities when examining the causes of change. 



Class Cohort 

For the purposes of general program validation, we undertook 
the extra effort of studying two successive years of class 
cohorts to minimize the possibility of unseen cohort effects in 
our general conclusions about change (Nesselroade & Baltes, 
1979). The cohort variable is not interesting in itsert^, but it 
proxies for whatever events on a social level were influential in 
student selection of a year to enter college. The age range of 
our population and the volatile environment of the seventies and 
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eighties in the changing roles of women, make this issue 
particularly important for our women students, many of whom are 
first generation college students. 



Ti ne Series Design 

It is part of our language to speak of "the four y<»ars of 
college" as if all the students who enter in a given year (or at 
least all the persisters) complete the program and graduate four 
years later. But this has never been the case. Indeed, with the 
influx of "new" students and their multiple life commitments, the 
four year model is already for many institutions a minority 
pattern rather than a norm. 

This is an important advantage for both our longitudinal and 
cross-sectional studies. We administer our external measures at 
entrance, two years later, and again a year and a half later, 
rather than when a student is a frishman, beginning junior or 
graduating senior. Thus, the time at which students are assessed 
on external measures is held constant, while the number of 
semesters they take to make that progress can vary. 



Because Alverno students are credentialed for successful 
demonstration of their abilities, at successive levels of 
sophistication, we use three measures of progress. One is the 
accumulation of her demonstrated ' ability levels on her 
performance assessments; another'^ is the number of credits 
completed and the third is the record of semester hoqrs 
completed. The first two measures vary because the number of 
ability assessments offered by an instructor, as well as the 
number attempted and completed by each student, is different in 
every course. Tnis time-variant approacn allows us to use the 
variation among students in the tiire they toend in college, and 
their quantitative and qualitafivt; progress th'^ough the program 
as a basis for comparisoti. We can then explore such key 
questions as whether students who perform more successfully in 
this program also show more change on measures from outside the 
college. 



Thus, in our longitudinal design, we assess students at 
consistent time intervals (see Table 1). Perfonnance in the 
curriculum varies, so we can investigate changes as a function of 
performance in the curriculum. Many studies of college effects, 
whether cross-sectional or longitudinal, have assessed students 
when they are Freshmen and when they are Seniors. In contrast, 
we have assessed an entire entering class as tliey uegan their 
studies, and then have reassessed the same group two years later, 
and for a third time, about two years later. Most entering 
students will be new Freshmen, but many will have prior college 
credits and, in class terms, will be Sophomores or Juniors at 
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Table 1. 



f Design tor the Administration of Human Potential Measures and Student Pisrception 
Measures 'for I+ongitudinal and Cross-Sectional Studies of Student OatcMes 
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Norp L« Fleure 4 for overview of components of the progra... validation model with measures Student 
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entrance assessment, A typical student who entered as a new 
^reshman and attended r.egularly for two years might in fact be a 
first * semester Junior at second assessment, but another student 
might have entered Alverno as a Sophomore by standing, taken only 
two courses in the entering semester, not registered again until 
second assessment, and still be a Sophomore, Class standing may 
be different at the third assessment two years later as well. It 
is precisely the variability in attendance an'! performance over a 
specified period of time that we use to investigate claims of 
change effects for the learning process as a global entity. When 
appropriate, we do take advantage of the fact that our design 
approximates the beginning, middle, and end of a typical 
student's college career, or that the assessment intervals 
approximate the periods of ger*»ral education and pre-professional 
education for the typical student. 

Achievement Cohort 

In a performance-based curriculum, students must demonstrate 
successive mastery of the competence levels. Their record of 
performance in the curriculum is an indicator of level of 
achievement in the curriculum at any point in time. Students 
vary in the number of competence units they have achieved, and 
can thus be categor'^-ed as high versus low achievement. The 
effects of performance in the curriculum can be studied in this 
comparison. 



Matching Comparison Groups 
for Degree Completio^ 

A well known probleia with comparing groups of entering and 
graduating students in cross-sectional studies is that entering 
classes include many students who will not persist through 
college, while a gradi»at_ng group consists of persisters by 
definition. Many studies try to control for this difference by 
matching students on some variable believed to predict 
persistence, most often an academic achievement variable. 

We were able to control directly for persistence in the 
cross-sectional study by using one of the entrance cohorts of the 
longitudinal study who were in college four years later, as our 
entering student comparison group. Thus, our cross-sectional 
comparison provides a conservative estimate of change. 



Controlling for Other Fa ctors 
Tha t Contr" ut e to Change" in" 
Perf ounance 

To relate change to p rformance in the learning process a 
aeries of analyses were first conducted to control for other 




factors ?:hat contribute to change in performance before testing 
for performance effects. First, age and other background 
variables (religion, parent's education and occupation, high 
school grades, prior college experience, and marital status) were 
tested for correlation with entrance assessments. Those 
varifiDles that accounted for differences in entrance assessments 
were then further examined for relationship to change in 
performance between assessments. If any background variable 
accounted for change between assessments, then that difference 
was controlled in testing effects of performance. Similarl' , 
effects of program differences incidental to the learning process 
(entrance cohort, residence, part time or full time status, and 
major) were .tested after background variables but before testing 
fot performance effects. Program variables accounting for change 
over that accounted for by background variables were also 
controlled before testing the relationship of change lo 
performance. Thus, any relationship between performance and 
change was only considered an effect of performance once the 
other possible sources of variance were controlled. 



Inct easing Rsl$ of Partictpation 

Readers experienced in longitudinal data collection will be 
interested in the extensive and detailed description ve provide 
of our data collection procedures (Mentkowski & Strait, 1983). 
We employed a range of effective strategies to get and keep the 
cooperation and participation of all studeats entering during the 
two-year entrance phase of the project. The continuous effort to 
motivate students to participate may be seen as reducing 
generalizability to populations not so motivated. We have taken 
the popition that complete data is a more important goal. We 
have motivated students to participate in order to achieve the 
highest possible rates of participation, and tQ stimulate the 
highest level of performance. 

In addition, we employed a variety of strategies to ensure 
the participation of professionals, particularly for those in 
management . These procedures are described in detail in 
Mentkowski, O'Brien, McEe^chern an<^ Fowler (1983). 
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Prouedures 

Procedures for carrying out the research objectives were 
designed to meet four broad objectives. 

• Create a context for validation research 

• Respond to concerns of students, faculty, 
and professionals 



• Collaborate with colleagues in research 
and curriculum development 

• Respect the values and objectives of the 
program and the research participants 

To carry out these objectives, we devised two strategies. 
One was to establi..i evaluation as a concept and function at 
Alverno College (Figure 2). The second was to develop more 
specific research methods that mirrored the values, objectives, 
and assessment process and techniques of the college, as well as 
the more recent frameworks and instrumentation in fields related 
to the study of college outcomes. 

Crea^d a Context for Validation Research 

We created a context for validation research primarily by 
establishing evaluation as a concept and function in the 
curriculum. As stated earlier, our colleagues in higher 
education were often interested in beginning 
evaluation/ val idation efforts of th^ir own. For this reason, we 
document the context for evaluation that we established at 
Alverno, as a case study that occurred at one institution. For 
us, this meant identifying evaluation goals for an Office of 
Research and Evaluation that could coordinate the functions 
necessary to carry out the research methodology and dissemination 
of the results. Consequently, this Office was created with the 
following overall goals. 

• Establish research and evaluation as a concept 
and function 

• Evaluate the quality, effectiveness and validity 
of the learning process 

• Contribute to creating a more goneralizable model 
of adult development and learning 

• Contribute to program development and student: 
development 

• Assist in insuring the quality of various 
research and evaluation activities vithin the 
college 

• Establish Alverno as an accountable educational 
institution in the community and as a contributor 
to higher education research and evaluation 

The Office of Research and Evaluation was established in 1976 
at the beginning of our efforts to validate the curriculum, and 
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is now a fully institutionalized and funded part of the college. 
The Office of Research and Evaluation was c eated as a service to 
the college. The evaluation/validation process is a curriculum 
component (Figure 2) that enables faculty to step back from the 
program objectively and systematically, and take a hard look at 
program functi^ni.ig -and validity iti- terms of^^udent aiid aiumnae 
outcomes* 



If research is to yield results with the broadest possible 
mplications, net only for the college, but also for the 
immediate community it serves, research personnel must consider 
themselves in service to the broader goals of their students and 
faculty colleagues. 

F aculty In ^ ^olvement 

The involvement of faculty is critical to the identification 
of the esearch questions, the carrying out of the studies and 
the critique and impl mentation of the results. Faculty were 
well aware of the concerns of higher education for validating 

outcomes, and the kinds of specific ?ssues and questions that 

were central to curriculum reform. 



Several ^faculty groups were essential to the validation of 
competences and assessment techniques. The Assessment Committee, 
consisting of members of the faculty who are specialists in 
assessment design, played a major role in designing and carrying 
out validation of assessment techniques. The Committee wcrked 
with the Office of Research ' and Evaluation to validate two 
interdisciplinary measures of college performance, and to create 

an instrument evaluation and revision process (Alverno College 
Assessment Comnittee/Of f ice of Research and Evaluation, 1980, 
1982, 1983). The Assessment Center provided structure for 
administering and scoring of the Integrated Competence Seminar 
using external assessors from the business and professional 
community. The Assessment Committee created definitions of the 
Six Performance Characteristics and in collaboration with the 
Office of Resear'^h and Evaluation, conducted the faculty rating 
of students on a measure (Six Performance Characteristics Rating) 
designed in the Office of Research and Evaluation to provide an 
external cross-disc ipl inary measure of college performance of the 
broad outcomes of col lege . The competence divisions » most 
notably Communications, Valuing and Social Interaction worked as 
research teams tc validate generic instruments (Friedman, 
Mentkowski, Deutsch, Shovar & Allen, 1982; Friedman, Mentkowski, 
Earley, Loacker & Diez, 1980). 



Respond to Concerns of Students, 
Faculty and Professionals 
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The Discipline Divisions, Chairpersons, Department 
Coordinators and all faculty members were involved in planning 
and carrying out strategies for involving students , for 
supporting the validation efforts, for communicating a rationale- 
for the studies to individual students or classes, for 
identifying ways to approach and involve students in followup 
efforts to enhance the number of students pat ticipating, and for 
providing opportunities for Office of Research and Evaluation 
presentations. These presentations to students were designed tc 
motivate students to participate, to give students feedback on 
their performance on one of *^^^ measures assessing human 
potential, or to give studeiics feedback on the overall 
evaluation/validation results. The Dean's Office and the 
Assessment Center collaborated co plan the administration of the 
human potential measures so that involvement and participation 
occurred as part of regular assessment procedures. 

Further, the Department of Business and Management and the 
Division of Nursing collaborated with the Office of Research and 
Evaluation to carry out the studies of professional perceptions 
an^'. performance in nursing and management, and used their 
credibility and networking in the community to establish contacts 
with individuals and organizations. Several ^offices in the 
college, having direct access to the business and professional 
c^BMMjnity^ -eontributed information and contacts for the studies 
of professional competence: the Office of Career Development, 
the Office of Off-Campus Experiential Learning, the Development 
Office, and the President's Office. 



Student Involvement 

We were also responsive to students' concerns and ideas. We 
surveyed att itudes^^rom half the student body in the spring of 
1977, and conducted in-depth interviews of the first ten 
graduates in 1976 (Mentkowski, 1977b). This information was 
particularly helpful in focusing some of the research questions 
iu our study of student perceptions. Issues identified for 
students included a focus on the relevance of education to 
careering aft**r college, the importance of improving the 
curriculum for all students, concerns with the validity of a 
newly formed program, interest in performing well after college, 
and so on. Student participants in the research often asked 
questions that helped us to clarify and focus the questions we 
were raising. These scudents also identified central concerns 
they had about participating so that adjustments could be made in 
data ccllection strategies. 

Involvement of Professionals 

Members of the business and professional community were 
already involved in the definition of competence (Advisory 




Councils), the citation and carrying out of experiential learning 
(through intern8hLp& aentored by professic.als in the field), and 
in the assessment process (through assessment designers in 
business, and external assessors of student performance dravm 
from the business and professional community). 
Evaluation/validation efforts relied on such external input • 

Collaborate with Colleagues in Research 
and Curriculum Development 



At the start of the NIE grant period, we were already 
collaborating with colleagues in higher education research and 
curriculum development, Alverno College had just completed a 
broad dissemination of the learning process through grants from 
the Kellogg Foundation and the Fund for the Improvement of Post 
Secondary Education, and many colleagues from other institutions 
had visited the campus over a period of years. We also 
maintained continuing relationships with instrument designers 
(see "Instruments") and centers that were working toward issues 
similar to those in which we were involved, such as the Center 
for Moral Education at Harvard University, McBer and Company of 
Boston, the Center for the Application of Developmental 
Instruction at the University of Maryland, and Loevinger's 
research team at Washington University in St • Louis • 

Early on, as a member of a consortium coordinated by McBer 
and Company and funded by FIPSE, we helped test new measures of 
college outcomes. We worked with members of American College 
Testing and the Educational Testing Service, who were interested 

developing innovative measures of college outcomes. At the 
same time, we were drawn to members of the research community who 
were experimenting with new measures of competence and strategies 
for assessing competence, as well as theorists in cognitive 
development and their colleagues who were measuring patterns in 
human cognitive growth. We identified an Evaluation Advisory 
Council made up of experts from other institutions who could 
provide more specific technical assistance. Our Evaluation 
Advisory Council (Donald Grant, University of Georgia; Milton 
Hakel, Ohio ^tate University; Joel Moses, AT&T^ assisted us in 
many issues related to design, instrumentation and validation 
during several visits to the campus. 

A major contribution was mide through extensive discussions 
on the issues conducted by Jean Miller of the National Institute 
of Education. These discussion meetings involved directors of 
five other projects. These sources, together with experts from 
our own faculty, formulated the more specific questions. Alverno 
faculty were part ot the research teams. This was important 
because they would be primarily involved in tryout, 
implementation and dissemination of results. The overall process 
ci ensuring responsiveness to the work by students and faculty 
contributed directly to the quality of the work. 7. us, the 
methodology was constantly under critique and review. 

(0 



Respect the Values and Goals 
of the Program 



Use Existing Program 
Evaluation Structures 

3ince evaluation/validation is a program component (Figure 
2), it necessarily follows that the methodology it carries out 
needs to be consistent with^he object ives and me thods of the 
other program components. Values underlying methodology need to 
be consistent as well. For us, several informal, nonsystematic 
processes for program revision and evaluation were already built 
into the program at the time the validation research was begun. 
For example, faculty understand very well what student 
perceptions and attitudes toward the curriculum are. In a 1977 
study of student attitudes where half the student body completed 
a survey (Mentkowski, 1977a), all faculty individually completed 
the survey the way they thought students would. Faculty 
accurately predicted modal student attitudes toward the program, 
toward faculty, and toward educational and administrative 
services (Mentkowski , 1977b) . In this case , we tested the 
informal ne.work for evaluating student attitudes in a more 
systematic way and demonstrated its effectiveness. This informal 
evaluative network is critical to planning strategies for 
involving student participants in validation research. 

We relied on systematic, in*-place evaluation and revision 
processes to carry out the research objectives. kti - Example of 
these processes is the one established to ensur e* ^esipn-based 
validity of the program. It includes regular review, evaluation 
2nd revision of assessment techniques established by the 
Assessment Committee for the faculty. These reviews figured 
heavily in the design of strategies for validating assessment 
techniques. Competence and Discipline Divisions also play a role 
in program evaluation and members of these groups served on the 
research teams for several of the studies. 

Provide Feedback for 
Program Improvement 

In an educational environment, validation is more a manner of 
making incremental and qualitative judgments than of making 
5^tatic and quantitative ones. It is interesting, after all, to 
know that students during a five year period demonstrated certain 
patterns of development and that these seem attributable to 
certain elements in the experience of "the college." But by the 
time we can make that statement, it is old news. 

What everyone really wants to know is whether today's or 
tomorrow's version of "the college" i<? likely to have similar 
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impacts on today's or tomorrow's students. Validation studies, 
properly designed, can help. They can enable us to make 
incremental judgments about whether and how the college is 
maintaining and improving its effectiveness in delivering 
desirable outcomes, as it evolves to meet the needs of subsequent 
cohorts and moves into new program are&s. 

Nor does validation simply stand aside and judge the 
college's evolution; -it contributes direct ty — to the — faculty's 
attempts to improve programs. The traditional concept of 
objective detachment is impossible from the outset, since the 
faculty's active collaboration is needed in defining and 
redefining outcomes, as well ab in devising at least some of the 
means (and arranging most of the opportunities) for measuring 
studpnf attainment of them. 



Attempting later in the process to "protect" the program or 
study participants from the impact of validation results would 
clearly be unethical, since qualitative feedback toward improved 
performance is the prime motive for both faculty and student 
participation. It would also be self-defeating. Closing off 
dialogue with the practitioners would immediately undermine the 
study's own validity, since it would cut off the primary source 
for making adaptive changes in validation methods and strategies. 
It would also lead, in practical terms, to disaffection and rapid 
termination . 



Instead of mutual detachment, the operating gode in 
validating a dynamic educational program is mutual collaboration. 
Joining with the validation team to interpret even early results, 
faculty then apply what they have learned and attempt to improve 
their program. The past becomes, in effect, the control and the 
present is an experiment in incremental change. If program 
modifications in turn yield improved outcomes, then the 
validation effort is itself validated along with the faculty's 
efforts. In a constant dialogue characterized by ongoing 
feedback and collaboration, practitioner and validator thus help 
each other to sharpen their focus, deepen their understanding, 
and improve their effectiveness. At certain points, our methods 
and results can be set forth for review by our Advisory Councils 
made up of experts in validation who serve as other, more 
external sources of critique and input. 



Contacting Participants 

A central concern in involving all participants in the study 
was to ensure that contacts with participants and organizations 
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met standards for involvement, including in^^rmed consent, 
confidentiality and feedback on the results of he studies as 
they became available. We were conducting longitudinal research 
with students and alumnae. Ineffective procedures could doom the 
project from the start • Further, som^ of the research was 
designed to build a bridge between the college and alumnae, and 
between the college and the professional community it serves. 
Our contact procedures, by communicating our efforts, could be 
expected to positively contribute to the reputation of the 
college and the degrees it offers. 

We consulted various members of the faculty and students in 
identifying strategies for contacting students. We also involved 
members of the college, Board of Trustees, Advisory Councils, and 
members of the business and professional ^ community to identify 
the most appropriate ways to contact professionals in nursing and 
management. We intended that procedures would respect 
professionals* right to informed consent and that would respect 
the protocol, administrative structures and values of the 
institutions and organizations who employed them. 



Communicat ing Rat ionale 
and Confidentiality 

In consulting with fi*culty and students, we identified the 
best ways to inform students who were research participants 
(Mentkowski & Strait, 1983). Prior to each assessment in the 
longitudinal and cross-sectional studies, ve made presentations 
on a regular basis to the faculty, informing them of upcoming 
student involvement, ways it would impart their class schedules, 
and the rationale for the studies so that they would be able to 
respond to student questions about the nature of their 
involvement. We consistently madft presentations to the students 
in classes about the rationale f their participation. Students 
were contacted individually if group presentations were not 
workable. In order to maintain student confidentiality of 
participation in the interviews of student perceptions which 
involved a subsample of students, all contacting was completed by 
private letter or by research staff who maintained 
confidentiality. ^'hen whole classes of students were 
involved — and who was involved was public knowledge — faculty were 
consulted about our procedures and their affect on individual 
students, who for personal reasons, were not participating. 
Great care was taken to involve student participants in the 
rationale for the study. On rare occasions when a student 
refused participation, such refusal was of course respected. 
Confidentiality of individual performance was maintained 
throughout the studies through a system of code numbers to which 
only research staff had access. 
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Student input in devel oping procedures for contacting and 
involving students in the work was particularly helpful (see: 
**Questions and Answers about Evaluation Studies: Third Report to 
Participants in a Longir.udin&l Sfrudy of College Outcomes 
Mentkowski , 1979) . Students critiqued study procedures and 
offered alternative suggestions that would assi ~ in devel )ping 
more effective strategies . 

We also took care in the involvement*, informed consent and 
confidentiality of professional participants. Oi.'ganizat ions and 
their executives were contacted initial ly with attention to 
rationale and informed consent. Participants were contacted by 
executives or administrators, and by the researchers with a 
rationale asking for participant consent. Confidentiality was 
promised and carried out (Mentkowski , DeBack, Bishop, Allen & 
Blanton, 1980; Mentkowski, O'Brien, McEachern & Fowler, 1982). 

Feedback on Study Results 

We made efforts to provide feedback on the results as they 
bec£nne available. Students involved in the studies received both 
individual and group feedback on the resul ts , in oral 
presentations and written reports (Mentkowski, 1981a, 1981b; 
Mentkowski & Fowler, 1981). Throughout the four years of her 
participation, and as an alumna in the followup studies, each 
student participating in the longitudinal studies received 
cpnsecutive, individual feedback and interpretation of her score, 
and group results on one of the measures she completed 
(Hentkowski , 1981a). 'ihe Chairperson of the Division of Nursing 
made a series of presentations to professional groups on the 
refi^ults of the study of the performance and perceptions of 
nurses , and copies of the report were distributed to the 
institutions involved. Copies were also distributed to attendees 
at dissemination sessions. A final report summary was mailed co 
each\ organization executive and manager participating in the 
stud> of management professional perceptions and performance 
(Mentkowski, O'Brien, McEachern & Fowler, 1983). 

Rate of Pgyt ic ipat ion Achieved 

We found our efforts to involve faculty , students and 
professionals and their organizations to be very successful. 
While such attention to creating procedures involved a large 
amount of staff time during the data collection which occurred 
over a five year period from 1976 to 1981, and added to the tiae 
and costs of the research effort, such effort was rewarded in 
high participation rates. First, student participation rates 
ranged from S3 to 99 percent across the three separate 
longitudinal assessments oyer a five year period (Mentkowski & 
Strait, 1983). Participation rates for the student perceptions 
study (Much & Mentkowski, 1982) were overall, 99 percent. 
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Eiglity-nine percent of the alumnae contacted *two years after 
college participated (Mentkowski, Much & Giencke«Hul 1 , 1983). 
Al 1 three insf itut ions involved in the study of nursing 
performance participated when contacted, as did 100 percent of 
the nurses invited to be interviewed (Mentkowski, DeBack, Bishop, 
Allen & Blanton, 1980). Fifty-three of the 55 organizations 
contacted for the management study agreed to participate as did 
94 percent of the managers and executives contacted (Mentkowski, 
O'Brien, McEachern & Fowler, 1982). 



The research described in this report was conducted over a 
period of five years. The fact that we could continue our work 
with students in the community of a small college and in the 
larger professional community over a long period of time and 
initiate and maintain participation is support for the 
effectiveness and ethics of our procedures. 



Chocsing, Ct^ating, Validating 
and Scoring Instruments 

Characteristics of Instruments 

In the section "Defining and Assessing Outcomes at Alverno 
College," we discussed the importance of selecting frameworks and 
measures for validating outcomes that match, as nearly as 
possible, the goals and assessment theory of the Alverno faculty. 
Instruments that we chose or created for each of the several 
research objectives were derived from curricuiar objectives, 
principles of assessment, and characteristics of assessment 
techniques that hav- been identified by the faculty. For us the 
faculty's definition of competence as developmental, holistic and 
generic , and the priuciples of assessment (Alverno College 
Faculty, 1979) arc a cornerstone in choosing and creating 
inctruments . 



Our validation insttu^.cnts must reflect the ge.ieral 
characteristics ot the faculty's techniques for assessing student 
performance if we are to adequately validate student outcomes. 
We must resist the temptation to import ready-made instruments 
currently available to assess outcomes and simply adopt thetn as 
validation tools. 



A program evaluation instrument, like an assessment 
instrument, should have the following characteristics. 

• It measures the learning objectives for a 
competence level or the broad ability being 
studied 



• It elicits the full nature of the ability—a 
holistic process 

60 



• It allows an opportunity to integrate content 
at an appropriate level of sophistication 

• Tt allows measurement of the integration of 
a competence with other relevant abilities 

• It is designed as a production tnsk rather 
than a recognit ion task 

• Its mode is similar to the ability as usually 
expressed, rather than an artificial mode 

• It will most likely be 'subjectively scored, 
by more than one assessor, against objective 
criteria ^ 

• It c be administered externally to the 
lear.j.ng situation — for example, in the 
Assessment Center 

• It is diagnostic, because the student expects 
structured feedback as an intrinsic part of 
every experience in v^ich the college asks 
"er to demonstrate her abilities 

• It provides evidence for credent ial ing the 
student's performance (Mentkowski, 1980) 

While performance-based curricula are likely to emp.oy 
criterion-reference^ measurement techniques , Alverno' s 

student-centered curriculum also creates measures that elicit a 
range of individual differences to provide adequate ^'nformation 
on the unique way each student demonstrates her abilities. Such 
information is particularly useful for diagnostic student 
feedback. Thus, instruments may also be designed to measure a 
range of student performance in meeting criteria as well as to 
provide evidence that the student was or wfis not credent ialed . 

While production type tasks usually generate qualitative 
rev^ults, we rely on generating both qua lit at ive and quant i cat ive 
data for responding to the range of research questions. The 
instruments we selected or created for measurement of each 
component of the validation model are indicated in Figure 4, 

Types of Measures 

Recognition and 
Production >teasures 

Two forms of organization were explicitly built into the 
selection of instrunents: i:he production versus recognition 
characteriat ic and the developmental continuum characteristic. 
Both characteristics stem froir Alverno assessment theory. 
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The task rharac uerist ic of production versus recognition has 
been given a thorough t- eatment by McClell ind (1980) though he 
refers to them as "operant" versus "respondent" measures. The 
basiCc issue* is that, across many kinds of research questions, 
instrument tasks that in some way ask the participant to respon^^ 
in the terms of trie te<Jt developer rather^fthan create or jr^oduce 
a resporse,' have been poor predictors of future behavior cT the 
person. Recognition measures test the 'investigators' r?w^ ity, 
^ut pot necessarily the reality of the participant, and it is 
usually the participant ve want to know something about. While 
this perspective puts a higher value on operation or production 
measures, a more neutral view would still hold that the two types 
of measures assess different things, so there is more to learn by 
using bath types of measures. We have intentionally used both 
production and recognition measures. 

"Because our criteria for instruments demand proactivity and a 
feel fc-generated response' on the part of the student, we have 
chosen the interview as the most unstructured production task. 
Interviews that measure student percept ion 53 liGSd to be derived 
from a student's thinking, and we have designed our own 
interviews for use with Alverno students and alumnae. Interviews 
with professiojial s have followed a standard critical-incident 
technique (McClelland, 1978) that is part of Job Competence 
Assessment (Klemp, 1*978), as has our instrumentation measuring 
learning to learn (Experiential Learning Log) (Mentkowski, 
O'Brien, Cleve & Wutzdorff, 1983). 

r , 

Developm ental Measures 

Some of our measures were designed by developmental 
psychologists to explore developmental phenomena, but. others were 
designed for other purposes. Our beginning assumption is that 
performance on every m^^sure has a developmental component, but 
that there are aspects of the 'task iitiich are affected by 
non~developmental experiences and abilities. 

External Cyitcrion Measures ^ 

A or task * this rese^jrch was to select a battery of 
external criterion measures (Human Potential Measures) . Measures 
oi college outcomes have come under fire as- measuring knowledge 
without performance, and as unrelated to future performance after 
college (McClelland, 1973). In fact, we>have not been able to 
identify any one extrrnal criteri' i measure th^t provides a 
perfect match to any of :he abilities we are validating. Given 
our criteria for instrument characteristics, particularly that 
they should be production tasks in order to measure the learner 
in action, few measures meet either the d**Tnands for the holistic 
nature of the ability or the mode of meat ^i.ement . We have found 
that internal validation is best carried out with faculty 
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designed^ generic measures: generic instrutents measuring Valuing 
tfnd Communications (Friedman, Mentkowski , Earley, Loacker & Diez, 
1980), Social Interaction (Friedman, Mentkowski , Deutsch, Shovar 
Allen, 1982), the Integrated Competence Seminar (Alverno 
College Assessment Comirittee/Of f ice of Research and Evaluationi 

1982) , and the Six Performance Characteristics Rating (Alverno 
College Assessment Co -^ittee/Of f ice of Research and Evaluation, 

1983) . 

External validation is most e£fective with measures of broad 
outcomes ( cognitive dpvelopment , ^ earning styles and generic 
abilities). As external criterion measures, we selected 
instruments that assess broad outcomes. Some 
cognitive-developmental measures, and recently developed measures 
of f^neric abilit-.es and learning styles (e.g., the Cognitive 
Compei^ence Assessment Battery developed by McBer and Company), 
have more nearly met our criteria for inst rumePts , and allow us 
to "talk to" researchers and theorists outside the college 
through the common language of test scores and quantitative 
resul ts . 



One advantage of usin^ criterion measures tihct have achieved 
some reputation is that other colleges are also participating tc 
some extent in collecting data on students. As members of a 
consortium of colleges, we cooperated with McBer and Company who 
have administered many of the instruments used in this study to 
students at a range of colleges and universities with both highly 
selective and more open admission practices (Winter, McClelland & 
Stewart, 1981). James Rest (1979a, 1979b) maintains a 
clearing ou^e on Defining Issues Test data and Jane Loevinger has 
published college student norms on the Sentence Completion Test, 
v^ich are useful in comparing changes of Alverno students with 
those at other colleges. Other norms for the Perry scheme 
(Mines, 1982) and Kohiberg's stages are also available (Kohlberg, 
1981b) . 



In addition to the instruments that employ production type 
tasks, we use a variety of ratings and questionnaires. The Six 
Performance Characteristics Rating, the Attitude Survey, the 
Management Perform;ince Characteristics Inventory, and the Student 
and Alumna Careering Questionnaires are examples of these types 
of measures. Student registration and assessment records provide 
/ a tange o^ information on student progress through the curriculum 

and performance on generic assessments. 

Create and Validate Instruments 

We had heavy involvement in creating and validating some of 
the instruments we used. Except for those instruments created 
for the nursing and management studies, most of these instruments 
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are designed to provide measures of college performance, 
attitudes or perceptions. The following instruments were created 
for the specific purposes of this study: 



• 


Six Perfonnance 


Characteristics Rating 


• 


Alverno College 


Attitude Survey 


• 


Alverno College 


Student Perspectives Interview 


• 


Alverno College 


Alumna Perspectives Interview 


• 


Alverno College 


Student Careering Questionnaire 


• 


Alverno College 


Alumna Careering Questionnaire 


• 


Management Performancji Characteristics Inventory 



• Management Careering Questionnaire 



In addition to this instrument development work, we olso 0 
realized that the state of tl^e art in developing college outcomes 
measures was such that we could expect to either contribute to or 
work toward the. validation of the instruments we were using as 
external criterion measures. Firet, we validated both the 
Integrated Competence Seminar (Alverno College Assessment 
ComTiiittee/Cffici of Research and Evaluation, 1982) and the Six 
Performance Characteristics Rating (Alverno College Assessment 
Committee/Office of Research and Evaluation, 1983). The Alverno 
Cv'lege Attitde Survey (Mentkowski, i977a) was tested for 
reliability and different forms were created for students in each 
of two major programs, and one type of scaling was compared 
against another (Mentkowski & Doherty, 1979). The Alverno 
College Student/ Al umna Perspectives Interviews (Mentkowski & 
Much, 1980a; 1980b) were revised. The student interview was 
created after initirl pilot work where students were interviewed 
with an even more unstructured open-ended interview (Mentkowski, 
1977b). The alumna interview was revised following the ^irst 
five interviews to clarify the questions. 



1l\ reg>.-d to the Human Potential Measures, we conducted an 
extensive valiiation of the instrument measuring the Perry ;cheme 
(Mentkowski, Moe«er & Strait, 1983). We carefully described our 
procedures for establishing and maintaining tlie reliability and 
validity of the ratings for the Se itence Completion Test of ego 
development (Mentkowski, Miller, Davies, Monroe & Popovic, 198:0. 
We collaborated with McBer and Company on the Cognitive 
Competence Assessment Battery by exchanging data, so that both 
Alverno and McBer had access to the most up-^to-date information 
on the validity of the measures. And we collaborated with David 
Kolb and Glen Gish on the valic' "ion of the Adaptive Style 
Inventory. 
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Score Instruments 



We employed three general strategies for ensuring the 
validity o: our instrument scores. First, we trained the 
administratoi s of the instruments, and kept extensive records on 
the procedures for administration so "^hat there would be 
comparab il it y in the administrat ion of the instruments , 
especially over the five years of a longitudinal study 
(Mentkowcki & Doherty, 1977, 1979, 1980d, 1980b). We also 
provided our interviewers with extensive training, and also 
trained our coders of qualitative interview data (fintkowski, 
DeBack, Bishop, Allen & Blanton, 1980; Mentkowski . O'Brien, 
McEachern & Fowler, 1982). Further, we consulted with instrument 
designers and their colleagues ^or all but one of the instruments 
(Watson and Glaeer, 1964), and participated in workshops that 
provided training by Lee Knefelkamp, for the coding of data from 
the Measure of Vocational, Educational and Personal Issues 
(ffeasure of Intellectual Development), for the Behavioral Event 
Interview by George Klemp and David McClelland, and for the 
Sentence Completion Test by Jane Loevinger. 

We knew from che outset that using production type tasks as 
college outcomes measures would require a large effort in the 
scoring and/or coding of the instrument". In order to accomplish 
this task, we used expert scorers outside the college for 
assistance. Scorers at McBer and Company directed by Ann Litwin 
comple^ed scoring of the Analysis of Argument, T**st of Iheraatic 
Analysis, Picture Story Exercise and Life History Exercise, and 
initially for the xest of Cognitive Development (see VJinter, 
McClellapd a- Stewart, 1981, for details). Hie Moral Judgment 
Instrument as scored by John Gibbs and Clark Power f om the 
Center for Moral Education at Harvard University. The Measure of 
Vocational, Educational and Personal Issues (af'^er Perry) was 
scored at Alverno, initiated by a workshop from Lee Knefelkamp 
and further input from William Perry, since Alverno was engaged 
in an extensive validation of thfe process for judging student 
performance on the larry scheme (Mentkowski, Moeser & Strait 
1983). The Sentence Completion Test of ego development was also 
scored at Alverno with input from two scoring workshops conducted 
by Jane Loevinger at Washington University (Mentkowski, Miller, 
Davies, Monroe & Popovic, 1982 ). 

T\\e Learning Style Inventory and the Watson Glaser Critical 
Thinking Appraisal were scored by hand «gt Alverno. The Defining 
Issues Test and the Adaptive Style Inv^tory were computer scored 
at Alverno with progra..is provided by James Rest and David Kolb, 
respectively . Alverno scored the performance characterist ic s 
inventories for the management and nursing studies with 
consultant assist ince from George Klemp and David Nk:Clelland, and 
Alverno coded data from the careering questionnaires. 
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Throughout the work, we experimented with various ways to 
analyze the open-ended interview data from tV^ study of 
student/alumna perceptions. Our methods ranged from creating a 
codebook specifying developmental levels of categories and 
example?, to a detailed analysis using all relevant parts of the 
data related to a question or category, to reading selected 
interview examples and generating a description of the overall 
findings. An outcome of this work is that we confirmed that an 
in-depth analysis of the material required a social science 
background in qualitative data analyp's. 

During the course of scoring the instruments, we created two 
detailed assessment processes and did extensive work to ensure 
their validity. One was created in collaboration with Lee 
Knefelkamp for scoring essays for the Perry Scheme (Me.itkowski , 
Moeser & Strait, 1983), and on.^ process was created in 
collaboration with George Klemp for deriving competences from the 
Behavioral Event Interview from the nursing study (Mentkowski, 
DeBack, Bishop, Allen & Blanton, 1980) and for coding the 
Behavioral Event Iiterview (McBer and Company, 1978) for the 
management study (Mentkowski, O'Bri n, McEachern & Fowler, 1982). 

Throughout the scoring and coding of all the data, we 
maintained contact vith the instrument designers with one 
exception (Watson and Glaser). We recognized that we needed to 
keep up to date with the latest information on the validation of 
the instruments, but more important, that the measures themselves 
s^ould benefit from 'he results of a five year longitudinal ptudy 
employing them. These results can greatly enhance our 
understanding of the meaning of the instruments because they wexe 
given as a battery and could therefore be interrelated. This is 
particularly important since many of the instruments are just 
being developed. In f'idition, we can piovide data on women's 
abil it ies . 



Another outcome of this extensive work is thi*t we have been 
able to disseminate some methodology useful to educators. The 
criteria and process used to judge student performance on the 
Perry scheme (Mentkowski, Mot ir & Strait, 1983), the Behavioral 
Event Interview process (Mentkowski, DeBack, Bishop, APen ct 
Blanton, 1980; Mentkowski, O'Brien, McEachern & Fowler, 1982), 
and the strategies for interviewing students have all been 
effective in various other projects here at Alverno (Schall & 
Guinn, Note 3), and some other campuses. 

Select Data Analysis Strategies 

Data analysis strategics were chosen following reviews of 
available methodology, James Rest and Mark Davison of the 
University of Minnesota and Klarcus Lieberman of the Center fr.r 
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Moral Education provided us with several suggestions and insights 
useful in the analysis of the longitudinal data from the Human 
Potential Measures. George Klemp guided our analysis of the data 
from the studies of professional competence in management and 
nursing. Finally, our Evaluation Advisory Council, Donald Grant 
of the University of Georgia, Milton Hakel of Ohio State 
University, and Joel Moses of AT&T, aided us in the validation 
and development of our college performance and performance 
characteristics measures (integrated Competence Seminar; Six 
Characteristics Rating). 

instniment Descriptions^ 

A brief description of each set of instruments is given 
below. Research reports describe the instruments in more detail. 

^!(iman Potential Measures: Cognitive Development 



Test of Cognitive Development 

(Renner, et al . , 1976; after Inhelder & Piaget , 1958) 

By having a student work a series of problems aad provide 
reasons for answers, this instrument measures a student* s 
cognitive activity based on* Piaget *s stages of cognitive 
development . The measure is more narrowly focused on a single 
stage of cognitive development, formal operations. 

Sentence Completion Test (Loevinger, 1976; Loevinger, 
Wessler & Redmore , 1970; Loevinger & Vtessler, 1970) 

A production task elicits a measure of an individual's stage 
of ego development. Ego her. is defined as one's style of life, 
the unity of personality, individuality, the method of facing 
problems, opinion about one's self and the problems of life, and 
the whole attitude toward making choice<^ iii all life spheres 
(Loevinger & Knoll, 1983). 

Moral Judgment Instrument (Kohlber'> , et al . , 1978; 
Kohlberg, 1981a, 1981b; Colby, et al . , in press) 

This production task elicits response to a moral dilemma. 
The ins»-ruraent provides a measure of an individual's stage of 
moral development by analyzing the reasoning a person gives in 
resDonpR to ques^'ions that probe reasoning about moral issues and 
orientations that create and define moral dilenonas. 



"^Sources for information on instrument availability, 
administration and scoring are found or4 page 85. 
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Defining Issues Test (Rest, 1979a, 1979b) 



Rest's instrument (based on Kohlberg's theory of moral 
development) provides a measure of an individual's moral 
development in a recognition task by analyzing the relative 
importance attributed by >i person to principled moral 
considerations. A person attributes importance to several 
reasons given for resolving a particular moral dilemma, and then 
rank orders them. 



Me fiSure of Vocational, Educational, and Personr^l Issues^ 
TKnefelkamp, 1974; Widick, 1975; Revised for Knefelkamp & 
Slepitzd, 1976; after Perry, 1970; now titled the 
Measure of Intellectual Development ; Mines, 1982) 

This production task measure of the Terry scheme of 
intellectual and ethical development asks students to write three 
essays describing "tne best class you've taken...,' "a decision 
about something that had major import ance ..o ," and ''things you 
consider ^en approaching the question of career choice... ." The 
essays are judged for position on the Perry scheme of 
intellectual and ethical development, using the Alverno Criteria 
(Mentkowski, Moeser & Strait, 1983). 

Human Potential Measures: Learning Styles 



Learning Style Inventory (Kolb, 1976) 
Ad aptive Style Inventory (Kolb, 1978) 

The Learning Style Inventory is a measure of individual 
learning styles which affect decision^making and problem-solving . 
The four styles are Concrete Experience, Reflective Observation, 
Abstr;ict Conceptualization^ and Active Experimentation- This 
recognition task requires the student to rank order descriptive 
statements about the mode of learning. The Adaptive Style 
Inventory measures the extent to ^^ich the person is likely to 
use each mode in various situations, and assesses for adaptive 
competeace through a recognition task. 

Life History Exercise (Klemp & Connelly, 1977) 

This instrument using a recognition task is a measure of 
interpersonal 1 eari^ing skills. The cases are programmed in such 
a way that a person with good judgment about people (i.e., one 
who does not make snap, impulsive judgments) will become more 
accurate in choices of the correct alternative as the respondent 
pro'^eeds through the case. Tlie instrunent assesses how one u'^^s 
infonn^sticn in making decisions about others or predicting 
behavior and examines the process by which ilecisions are made. 



Human Potential Measures: Generic Abilities 



Analysis of Argument (Stewart, ]977a) 

This instrument is intended to assess intel lec tuai 

flexibility by requesting the student to argue against a 

controversial opinion, and then defend the opinion just attacked. 
The measure uses a production task. 

Test of Thematic Analysis 

(Winter, 1976; Winter & >k:Clelland, 1978) 

This instrument consists of cwo sets of thr ie stories. 
Students are asked to compare the sefs thematically, a production 
task. This "thematic analysis" is scored according to twelve 
categories of critical thinking. This test is based on an 
understanding of cognitive development defined as the ability to 
analyze new information and to synthesize new concepts based on 
this information, and reflects the ability to integrate 
information into one's own cognitive structure. As the cognitive 
structure grows, so does the ability to think critically, to make 
a cogent argument and to reason inductively. 

Picture Story Exercise (Winter, McClelland & Stewart, 1981) 

Th^ instrument requires the student to write narratives to 
six pictures. This instrument, modeled on the Thematic 
Apperception Test (Morgan & Murray, 3935), is used to assess a 
variety of abilities. One is "sel f--def init ion" which, emcompasses 
the way one thinks about the world and one's self, the -^ay one 
reacts to new information, and the way nne behaves (Stewart & 
Winter, 1974). People with high cogni:ive initiative are not 
only able to think clearly, but also to rti.ison from problem to 
solution, and to propose to take effective action on their own. 
"his instrument is also u*)ed to assess Need for Achievement 
^(McClelland, et al . , 1953), Affiliation (Atkinson, 1958), Power 
(Winter, 1973> and Activity Inhibition (McClelland, 1975). 
Stages of Adaptation, a measure of ego development created by 
Stewart (1977b, 1982) after Freud and Erickson, are also scored 
from the Picture Story Exercise. 

Wat son~Glaser Critica] Thinking Appraisal (Watson & Glaser, 1964) 

This is a traditional '>nd time tested recognition task 
measuring several components Oi critical thinking. Inference, 
Recognition of Assumptions, and Deduction were used in the 
current study. 
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Measures of Student Performance 
in the Learning Process 



Academic Reports: Progress (Mentkowski & Strait, 1983) 

Several indicators of progress through the curriculw are: 
number of semesters attended; number of credits achieved; and 
number of competence level units achieved at any one point in 
time . 



External Assessments : Integrated Competence Seminar 
(Alverno College Assessment Committee/Office of Research 
and Evaluation, 1982) 

This assessment technique provides an opportunity for a 
student from any discipline to demonstrate integration of her 
abilities developed by the midpoint of her college education. 
The student shows how she transfers these abilities to a new and 
complex simulated situation she is likely to face as a 
college-educated person. The student is not credentialed On this 
instrument; it is used for diagnostic purposes only. Thus, the 
ICS is, in part, an ex^'ernal criterion measure of the student's 
ability to transfer her learning after completing the general 
education sequence. The student performs thfee c^xercises over a 
four hour period as a manber of a decision-mal^l ig board of 
citizens: Oral Presentation, In-Basket , and Group Discussion. 
Of f -cam pus professionals serve as assessors , who individually 
observe and evaluate each student's performance against specified 
criteria, come to consensus, prepare a written evaluative 
statement, and ^ ?t individually with each student to provide 
feedback on he. performance. This instrument* has since been 
rev ised . 



In tegrated Evaluation: Six Performance Characteristics Rating 
Tai verno College Assessment Committee70f f ice of Research 
and Evaluation, 1978, Revised 1979) 

Six cross-disciplinary performance characteristics were 
identified and defined by the faculty (Alverno College Faculty, 
1977) to describe the student's developing ability to interrelate 
and internalize performing the competences. At the present time 
the facultv have defined five characteristics whJch apply to her 
benavior — Integration , Independence, Creat ivijy, Awareness , 
Commitment — and a sixth — Habituality — which modifies the others. 
The developraent of these characteristics takes place initially as 
the student strives to acquire or to improve abilities demanded 
by her discipline or profession. Gradually, the characteristic^* 
themselves become central to her style of worki ig and to her 
exercise of personal responsibility. Faculty thiik of these 
characteristics as contributing to her personal and professional 
life, and incorporate assessment of them into a longitudinal 
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evaluation program. Each first and second year student in the 
Weekday College is rated on each of the charac teriggics by an 
instructor at the end of each year in college. Eaon^third and 
fourth year student receives two independent racings from two 
professors in her area of specialization and one rating from a 
professor in her minor area. 

Measures of Student/Alumna Perceptions 
of Learning and Careering 

Alverno College Student Perspectives Interview 
(Mentkowski & Much, 1980b; In Much & Mentkowski , 1982) 

This interview is designed to enable students to speak for 
themselves about their college experiences. It measures 
students' perspectives on many aspects of college learning. It 
is open-ended and probes students' thinking , asking them to 
describe their perceptions of learning, how they have changed, 
and v^y. Questions that focused the design of the interview 
questions are also included. 

Alverno 5PJ_^^^^_ j^^_"P^^ ^^y.^_^ 

'(Mentkovaici & Much," 1980a; In Mentkowski, Much & 

Giencke-HoU , 1983) 

Similar in form to the interview for students, this interview 
focuses on an alumna's perspectives on transfer processes, new 
learning, careering and professional development and integration 
of personal and n-ofesaional roles. The open-ended -questions and 
probes ask her ' • ^ ' relationships between college and her life 
as it is 'lov a questions are included. 

A l >' erno Col lege At t i t uH e ^ uryey 

TFoi n A'^nT FoiT B) (Mentkov-ki, 1977a) 

Ihe ^ titude Survey is an objective questionnaire measuring 
students' i^ercepwions of and attitudes toward a variety of areas 
such as: "Alverno," "Faculty," "Changes," "Syllabi," 
"Pe formanco," "Professional Development," "Assessment Center," 
"Competence Level Units," "Challenge," "Progress," "Social Lifj," 
"Advising," "Career Development," "Need for Structure," "Learning 
Modes," "Confidence," "Library, Student Activity, Residence 
Hall," and "General Issues." 



92 



Alverno College Student Careering Questionnair e 
(lientkbwski & Bishop, 1981; In Mentkowski, Much & 
Giencke-HoU , 1983) 

""his careering questionnaire is administered to gra4uating 
seniors and includes questions in an objective format that 
requests information on the following: paid am unpaid work 
experience before, Huring and after college; field experiences, 
clinical experiences, studert teaching and participation in 
'^iverno's Otf-Campus Experiential Learning Program; salary before 
college and now; expectations for getting a job and changing 
jobs; expected job area and title; expected salary range; ratings 
of college preparation . for future job performance; expectations 
of future job satisfactions and potential ^for advancement; 
expectations for performance on job-related exams; expectations 
for continued schooling; self-description of motivation to learn; 
rating of the effectiveness of the Alverno experience; 
satisfaction w^th current choice of major or career; expectation 
of advantage of a college, degree in getting a job; ratings of 37 
general and specific goals expected to result from college on 
three variables (goal importance, goal preparation from Alverno, 
and current goal achievement); parents* occupations, number of 
children, maritax stalis. Attitudes toward working and sources 
for financing college are also included in the revised 
instrument • 

Alverno College Alumna Careering Questionnaire 
(Mentkowski & Bishop, 1980a; In hfentkowski , Much & 
Giencke-Holl, 1983) 

This careering quest ionnf^ire includes the questions contained 
in the instrument for Alverno graduates, but it is worded to 
collect data on information such as work experience since 
graduation; success in obtaining a position related to a chosen 
career; current salary, salary increase; ratings of career 
preparation; potential for satisfaction and potential for 
advancement in the first position held after graduation; reasons 
for not seeking paid employment after graduation; performance on 
exams related to obtaining a job; continued education; 
satisfaction with college career choice; ratings of goals now; 
membership in career-related associations; and membership in 
civic or other organizations. 



Measures of Professional Performance, Perceptions, 
and Careering and Professio^ial Development 

Behavioral Evei Interyie w 

(Klemp, 1978; McClelland, 1976, 1978) 

This critical incident technique elicits six situations which 
tne professional identifies as effective or ineffective. The 
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professional, guided by the interviewer, describes esch 
situation, who was involved, and the outcome, but primarily 
focuses on 'what he or she actually did in the situation, and what 
he or she was actually thinking, feeling and intending at the 
time. Interview transcripts are used to create the Behavioral 
Event Interview Writeup from v+iich competences can be derived or 
coded . 



Behavioral Event Interview Writeuo (Alverno College Office of 
Research and Evaluation, 1980; after McClelland, 1978; In 
Mentkowski, O'Brien, McEachern & Fowler, 1982) 

Interview content f^'om the Behavioral Event Interview 
transcript is suramarized in written form as soon as possible 
after the interview. The form calls for a description of each 
incident, the components of the incident, the participant's job 
responsibilities, and the characteristics he or she thinks 
necessary for outstanding performance. For each incident, the 
participant's behavior is described in detail, and wliat she was 
thinking, feeling and intending at the time. The result or 
outcome is alsj described. Information that would identify the 
participant is deleted. 



Nursing NoTiiination Questionnaire (after Kiemp, 1978) 

The rom" latioti que*5tionnaire is a two-page instrument that 
bilefly describes the study and asks participants to list those 
profes£*inhal peers whom they consider to be "outstanding." Space 
is allotted for tv i names. Participants are asked to list as 
m^ny "outstanding" peers as they can from rorrery. 

Nursi ng Jpb Element Inventory (Grau ^ Rutter. 1977) 

Thr Job Element Inventory is compri^^^ed of a list of 120 
j3^ri^ornian(te characteristic.*? nurses identif^eo tis necessary for 
"oi^tstantJ'^'^-" or "superior" job per form/ i^^e . Ihe purpose of the 
inventory , to ascertain the ^ <ihaviors/characleristics 

participants think nurses must postjoS for outstanding nursing 
per form ance . Par tic i pan is re spond ir* the • ist three separate 
times. 'ITiey check those behavio^ - they believe (1) distinguish 
"outstanding*' from "good" nurs-s who share their job title, (2) 
characterize 'marginal" nur-.s who share their job title, (3) are 
more impotit-int in hirini^ .r training for their job. 

Nursing Careering Questionnaire (Grau, 1977) 

■■ — J — « — i. — 

In ol|jective format , the questionnaire el icits information 
about marital status, number of dependents, year of licensure, 
years of'nursing experience^ type of educational preparation for 
licensure, current educational pursuits, an estimate of future 




educational endeavors, job satisfaction, and self-evaluation of 
job performance. The measure was used to categorize interview 
participants on educational background, years nf nursing 
experience, hours of employment per week, marital status, job 
satisfaction and sel f~percept ion of performance. 

4. 

Management Per formancL Char ac ter ist ic s Inventory 

(Bishop, Mentkowski, O^Bripn, Birney, Davies & McEachern, 19P0; 

In Mentkovski, O'Brien, McEachern & Fowler, 1982) 

Each manager is i sked , to judge each of 162 statements or 
performance characteristics Ol ) as relevant to one's own work 
experience, (2) as essential to selection and trailing, and (3) 
as characteristic of outstanding performers. Characteristics 
that meet all three criteria for judgment are then considered to 
be descriptive of effective management performance from the point 
of view of the managers studied. Through an item scoring 
procedure, characteristics that discriminate average from 
outstanding performers are id*.itified. 

Mana|;ement Careerxng^ Questionnaire ( rie n t ko W64 i ^ b^' sha^, -l^SObi- 
In Mentkowski, O'Brien, McEachern & Fowl ci , 1982) 

Thi^ measure collects information on several variables 
relate to careering and professional development. .Infonaation 
is gathered from the manager on job responsibilities and 
function, who the manager reports to and who reports to tht^ 
manager, position level and type, experience in the company anc 
the last position the manager held in her previous company. Ihe 
manager also reports levels of education completed and in 
progress, ea of specialization, and coapletion of a management 
training program. Number and breadth of ptofession^l activities 
not sponsored by the company are also ind'cated. Personal 
information includes marital status, number of children, number 
of roles, husband's occupation, moto r's and father's occupatiot., 
parent*^' occupat^o.ial status, number of siblings, ^nd birth 
order. 



Instrument Sources 



Hi** description of each instrument is found on pages 7S-8^^ of 
the text. The following list can assist the reader in gaining 
information about instrisnent availability, administration and 
scoring . 



integrated Competence Seminar 
Six Performance Characteristics Rating 
Alverno College Student Perspectives Inter\'iev 
Alverno College Alumna Perspectives Interview 
Alverno College Attitude Survey 
Alverno College Student Careering Questionnaire 
Alverno College Alumna Careering Questionnaire 
Management Performance Characteristics Inventory 
Management Careering Questionnaire 
Nursing Nomination Questionnaire 
Nursing Job Element Inventory 
Nursing Careering Questionnaire 
Perry Scheme scoring criteria in Mentkowski , 
Moeser & Strait , 1983 

. Alverno Office of Research and Evaluation 
Alverno College 
3401 South 39th Street 
Milwaukee, Wisconsin 53215 



Learning Style Inventory 
Life History Exercise 
Analysis of Argument 
Test of Thematic Analysis 
Picture Story Exercise 
Behavioral Event Interview 
Job Competence Assessment 

McBer and Company 

Test and Scoring Division 

137 Newbury Street 

Loston, Massachusetts 02116 



Moral Judgment Instrument 
(in Colby, et al . , in press) 

Lawrence Kohlberg 

Center for Mor&l Development and Educat ion 
Harvard University 
Larsen Hall, Appian Way 
Cambridge, Massachusetts 02138 



Defining Iss les Test 
( in Rest , 1979 a and b) 

James Rest 

Minnesota Moral Research Projects 
330 Burton Hall 
178 Pillsbury Drive S.E. 
Minneapolis, Minnesota 55455 
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Measure of Vocational, Educational, and Personal Issues 
(now called the Measure of Intellectual Development) 

Lee Knefelkamp 

Counseling and Stuc\?nt Personnel Services 
College of Education 
University of Maryland 
College Park, Maryland 20742 



Learning Style Inventory (see McBer above) 
Adaptive Style Inventory 

David Kolb 

Oepartment of Organizational Behavior 
Sears Library Building 
Case Western Reserve Library 
Cleveland, Ohio 44106 



Wateon-Glaser Critical Thinking Appraisal 

Harcourt, Brace, Jovanovich, Inc. 

757 3rd Avenue 

New York, New York 10017 



Test of Cognitive Development 
John Renner 

University of Oklahoma 
School of Education 
Norman, Oklahoma 73069 



Sentence Completion Test 

(in Loevinger et al . , 1970 and Loevinger & Wessler, 1970) 
Jane Loevinger 

The Social Science Institute 
Box 1202 

Washington University 
St. Louis, Missouri 63108 



The Triangulated Validation Model 

The triangulated validation model has its roots in the 
strategy of investigation most disciplines use in the liberal 
arts, that of comparison among different perspectives and ways of 
knowing, rather than comparison between an experimental and 3 
control group. Thus, we aggregate findings from the same 



individuals using multiple measures and perspectives, use 
longitudinal designs where the individual is her own control, and 
compare results from at least three data sources or groups. This 
model demands in-depth studies in a single setting. Since this 
model represents an ongoing research program, not all possible 
f'ata sources, results or comparisons among sources are complete 
at this time. But enough are complete after seven years to 
provide a tableau of findings. 



The triangulated validation model is designed primarily to 
generate results that enable educators (1) to improve their 
practice and (2) to develop better theories of adult learning and 
life-span growth. We believe that the motivation to improve 
practice comes from a comparison between how a curriculum 
currently enhances student learning, the value added, and what 
educators know is still possible for a curriculum to achieve. 
Results from our validation model stimulate faculty motivation to 
improve, just as our student assessment model, with its emphasis 
on individual comparison of current performance against past 
performance, motivates the student to assimilate the feedback and 
ira prove . 



The triangulated validation trodel includes comparison between 
colleges. Indeed, we participated in a seven college comparison 
study conducted by Winter, McClelland and Stewart (1981) where 
instruments Alverno students completed were administered at other 
colleges. But our model does not limit such comparisons to 
norm-referenced measurement which has usually characterized 
comparisons among colleges. Rather, faculties exchange data on 
students, ways of teaching, and assessment strategies. The 
purpose of the exchange- is not to compare an experimental or a 
"control*' college or to compare colleges on some single variable 
or dimension of "low to high,'* poor to excellent. Rather, the 
design allows comparing and contracting among a multiplicity of 
data sources and faculty teaching strategies and assessment 
techniques . 



Student Outcomes 

We studied student abilities from multiple por'nts of view. 
At the first and second levels of the triangulated validation 
model we used a variety of lenses to capture a picture of student 
outcomes . 



A picture of student abilities emerges from facul ty- judged 
student performance on faculty-designed assessments. We enhance 
this picture by studying anonymous student perceptions of their 
learning. Student growth on measures of human potential drawn 
from outride the college give still a third picture of student 
outcomeb. Findings that are corroborated from all three points 



STUDENT OUTCOMES 



PERFORMANCE. 




PERCEPTIONS POTENTIAL 

of view are considered to have relatively more validity than 
results that emerge from only one or two. 

It, is also clear from the next illuotration, and the previous 
review of instruments, that within each of these three dimensions 
of student outcomes we used at least three kinds o f measurement , 
employing both recognition and production tasks. 

STUDENT OUTCOMES 



PERFORMANCE 




PERCEPTIONS POTENTIAL 
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To 8t udy per formance , we used ac ademic reports of student 
progress in the curr iculum , an external assessment (Integrated 
Competence Seminar) and an integrated ^v _ ^j^^^^^O" that consisted 
of a faculty rating of how students demonstrated their abilities 
(Six Performance Characteristics Rating). Our study of 
perceptions included a perspec t ives interview, a survey of 
attitudes , and a questionnaire of careering expectations and 
goals. We studied the human potent ial of our students from three 
separate theoretical frameworks: theories of 

cognitive-development , experiential learning theory and learning 
styles, and measures of generic ab i 1 it ie s where the abilities are 
expected to link education and work performance. 

We also studied abilities across multiple points in time. We 
recognized that we needed to create a picture of student outcomes 
that would result in a series of at least three photographs of a 
student's emerging aoilities. Thus, we created a cross-sectional 
and longitudinal data basc , conducting sets of parallel studies. 
The longitudinal results allow us to describe the patterns 
characteristic of intra-indiv idual development and the unique 
ways students develop these outcomes. 

Student, Alumna and Professional Outcomes 

We studied abilities using multiple groups. We wished to 
widen the lens, to examine the abilities of our student s as 
alumnae , and to study professionals who were not our students. 



STUDENTS 




PROFESSIONALS ALUMNAE 
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Similarly, we selected instrumentation that would enable a 
comparison of findings from at least three kinds of measures . 
Student assessments are described above. We studied alumnae 
perceptions through a perspectives interview, alumnae careering 
through a questionnaire, and inferred alumnae performance through 
the interview. Future studies will use independent measures of 
performance. Professionals * perceptions were studied chrough 
ratings of abilities critical for education, selection and 
performance. Professionals * careering was measured through a 
questionnaire. A behavioral event interview tapped critical 
incidents of effective and ineffective professional s ' 
performance . 

Research and Evaluation, 
Cun^iculum and Outside Sources 

Finally, we studied abilities such that we had multiple 
opportunities for critique and. comparison. We created a context 
for validation research by establishing evaluation as a concept 
and function in the curriculum. We responded to the concerns of 
students, faculty and professionals, consistently respecting the 
values and goals of the program and the research participants. 
Further, we collaborated with colleagues outside the institution 
in conducting the research. Figure 5 further r.eilects similar 
characteristics of our research efforts once the results are in. 
Figure 5 depicts all four levels of the triangulated validation 
model. The fourth level consists of the validation model from 
the research and evaluation component of the instructional system 
researching'^student , alumna and professional outcomes and depicts 
the opportunities for critique and comparison with other data 
sources from the curriculum and from out side 30 ur c es . 



Find ings from the study of students , al umnae and 
professionals gained through systematic research and evaluation 
are compared by the Office of Research and Evaluation staff and 
the faculty to the data generated from various sources in the 
curriculum . These include student performance on the full range 
of over one hundred performance assessments all students complete 
(as opposed to data from only those students who were 
participants in the research designs), data from regular course 
evaluations completed by students, and so on. Data from faculty 
and administrator review processes systematically evaluate 
student per formance and curriculum structures* The first five 
research reports emerged in par t from these internal eval uat ion 
processes ( Alverno College Assessment Committee/Office of 
Research and Evaluation, 1980, 1982, 1983; Friedman, Mentkowski , 
Deutsch, Shova)' & Allen, 1982; Friedman, Mentkowski, Earlev, 
Loacker & Diez, 1980). Data also obtained from external 
reviewers (external assessors of student performance, mentor s of 
students engaged in off-campus experiential learning, external 
advisors of discipline departments) can be compared to results 
from the research and evaluation source. 
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Figure 5. The triangulated validation model, 
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Data from tha research and evaluation sources and from the 
other sources in the curriculum can come up against <=urther 
critique from a review of peers and through comparison with data 
from similar instruments administered to other colleges in the 
research community . The accrediting process is another source of 
further data and critique . Perhaps the most dynamic and 
systematic critique and comparison occurs in the Alverno 
Institute, the institutionalized structure v^^ereby findings are 
exposed in presentation and dialogue to the critique and review 
of the higher education community . The section on dissemination 
in this report details the number of colleges t' have attended 
seminar days or week-long workshops where the> Q#ve received 
consistent reports of the progress aud findings of the research. 
Input and questioning by our learned colleagues creates a 
continuous dialogue, critique and review. 



The four levels in this triangulated model are our 'attempt to 
ensure that results emerge frou multiplicity in measurement and 
theoretical frameworks. Since the college is conducting its own 
research and evaluation, it is critical that we build externality 
into every level of the triangulated model . At the instrument 
level, multiplicity in measurement ensures breadth in qualitative 
and quantitative assessment; multiplicity in theoretical 
frameworks controls for bias in perspective and allows comparison 
between the faculty's own theories of learning and assessment and 
those drawn from outside the college. At the level of -student 
outcomes, measurement of performance and perceptions are 
comprised of Alverno-designed instruments; measures of potential 
are drawn from outside the college. At the third level of the 
validation design comparing student , alumna and other 
professional outcomes, professionals were drawn from groups of 
individuals who were not Alverno students or alumnae. At the 
fourth level, research and evaluation outcomes are compared to 
outcomes within the Alverno curriculum, but they are again 
compared with outside sources. Thus , multiple, external 
opportunities for critique and comparison are built in at ev.ery 
level of the triangulated model. 
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WHAT ARE THE OUTCOMES OF AN 
ALVERNO COLLFGE EXPERIENCE? 



Can an ability-based curriculum really make a difterencf? 
Can It prom.^rc t:r,e kina of J road personal and mtelioctunl 
development triat lasts a lifetime? Can it enhance a person's 
skills, and impiove one's chancej; at having an effective career? 
Can it benefit the "new" st uaenls--adul ts , women, minorities — as 
well as Cradicional students? Oo .he abilities learned show up 
o.'» the job? 

Because this is an ongoing project, consisting of 
interrelated studies, we do not anticipate one set of "final" 
results. Most of our studies are coi^t nuing and will continue 
for several years. What we can report, after seven years, are 
preliminary results in our major areas of inquiry. For this 
overview and summary, we discuss results from the ten research 
reports under two major headings, "What are the Outcomes of an 
Alverno College Experience?" and "How Do Alverno College Outcomes 
Relate to the World of Work?" 



We have also been able, alopg the way, to make some 
contributions to the newly developing field of educiti^nal 
program evaluation and to the repertoire of procedures f^r 
validating developmental outcomes. It is for this reason that 
this report has included a*^- overv.iew and summary of the research 
methodology as well as the rationale and results. In addition, 
our work seems to be offering some substantive support for the 
goals of outcome-centered, ability-based curriculum design. 

We have taken three independent approaches to measuring the 
meaning and development of the broad abilities the Alverno 
curricuiom is expected to foster. These are: (a) student 
per form a nee on college-designed ability measures within the 
curriculum (Alverno College Assessment Committee/Office of 
Research and Evaluation, 1980, 1982, 1983; Friedman, Mentkowski , 
Deutsch, Shovar & Allen, 1982; Friedman, Mentkowski, Earley, 
Loackcr v, Diez, 1980; Mentkowski, 0*Brien, Cleve & Wutzdorff, 
1983 ), (b) student percept ions of the reasons for learning, the 
process of leftrning, and its value for their own career and life 
goal^ (Mich & Mentkowski, 1982), and (c) student performance on 
twelve measures from outside the college describing human 
potent ial for growth in cognitive development, learning styles, 
and ^ neric abilities (Mentkowski & Strait, 1983), Results are 
based on longitudinal and cross-sectional studies with over 750 
women students aged 17-55. A core group of 200 of these make up 
the longitudinal participant group. Seventeen measures generated 
1 7, 500 responses; about 365 longitudinal interviews complete the 
student ^)utcomes data base. 
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[st udent outcomes 



PERFORMANCE 




PERCEPTIONS 



POTENTIAL 



We ha seen significant chinge in all three kinds of 
measures. These changes confirm that students do change in their 
developing cognitive patterns and abi' ities — whether these are 
defined by faculty'or by other practitioners and researchers — and 
that students' own perceptions and uses of learning change 
concurrently. Before we describe the major findings from the 
research reports, we wLli recap the distinctive features of the 
Alverno curriculum so the reader may review the context for these 
changes in student outcomes. 



In redesigning our curriculum in the early seventies, we 
biiilt in several elements based on experience in teaching and 
assessing students (Alverno College Faculty, 1976). First-, 
abilities that we identified were complex and included knowledge, 
skill, self-perception, disposition, attitudinal and motivation 
aspects. Second, abilities were defined to make them teachable, 
assessable and transferable across settings. The abilities are: 



Commm ic at ion s 

Anal ysis 

Problem Solving 

Valuing in Decision Making 

Social Interac tion 

Taking Responsibility for the Environment 
Involvement in the Contemporary World 
Aesthetic Response 



The a: ^rno Curriculum: A Recap 



Abilities 




ERIC 



/ 
/ 



Teaching Strategies 

Faculty designed teaching strategies to develop thee^ 
abilities across the curriculum within the regular course 
structure. In addition, the experiential dimensions of the 
curriculum were expanded. Students in each of rhe 20 academic 
and professional programs engage in sponsored off-campus 
experiential learning where transferring abilities learned in 
college is paramount. Classroom experiences likewise focus on 
irwolvement ia learning situations where simulations and other 
concrete experiences, as well as reflection, conceptualization, 
and action plans are tested. Because all students are required 
to demonstrate their abilities in successive, increasingly 
coipplex performance assessment?^ , the abilities for^ a kind of 
"core curricul'jm" within the traditional discipline-based one. 
Faculty integrate these process ab il it ie?- with the content of the 
d isc i pi ine s . 

Assessment 

Central to the curriculum is a complex at.sessment process 
whereby student demi5nbtration of these broad- ab il it ies is 
evaluated (Alverno College Faculty, 1979). Developing and making 
explicit the criteria for assessment is a faculty effort to 
define these abilities and make them operational. Thus, faculty 
create criteria or descriptive statements nhat give themselves 
and students a picture of the ability to be assessed. Abilities 
are then inferred fiom performance. The student, the faculty, 
and external assessors from the Milwaukee profess* al and 
business community use these criteria to evaluate student 
strengths and weaknesses in performance situations. 

Assessment thus defined and implemented becomes an 
opportunity for learning. The assessment process provides O 
mirror of vjhere the student is in her learning and what abilities 
she still needs to develop. Detailed feedback -i-s given and 
practice of abilities is structured. Because assessment focuses 
on the application of abilities, students learn to tie knowledf^^, 
theory, motivation, and self-perceptions to productive action. 
Each sf.dent demonstrates her abilities in ways similar to the^ 
way the ability is usually expressed (e.g.. Social interaction is 
assessed through group problem solving) . . A'isessment thus 
contributes significantly to learning because the student is 
required to demonstrate abilities in many courses ahd multiple 
^set t ings . 

Further, faculty make a concentrated effort to assist the 
student to r^fgnize her own perspective on learning and her 
learning styles, to negotiate this new learning enviromnent, to 
integrate her learning -across courses, and to link learning in 
college to her career and civic life. Seminars for all beginning 
students focus on these goals. 
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Elements of Abiiity-Based Learning 



Wiat is different about the way ability-based learning at 
Alverno is taught? Elements of student learning that were 
formerly ai>sumed to be learned from the broad effects of the 
residential college atmosphere or as an outcome of completing a 
variety of courses have been explicitly defined. Ihese core 
abilities are consistently fostered and assessed across the 
curriculum. For example students, including those who are 

not "joiners," who ^nly on "getting A's," or who leave 

classes not for the ^yta ur the editorial board of the school 
paper but for job and/or family responsibilities, must 
demonstrate interactive abilities in the curriculum. Both the 
research findings and our experience have confirmed that faculty 
cannot take for granted that even the experienced adult student 
has mastered interactive skills, their application, or their 
integration with other abilities. 



Not only is ability development consistently fostered across 
the Institution, but the faculty have or ^nized themselves to 
ensure that it happens. Take, for example, analytical ability. 
All our faculty have redesigned their courses to foster the 
development of this ability within the context of their 
particular disciplines. By infusing criteria for analytical 
ability into teaching/ learning activities,' assignments §jifi 
assessments, faculty create an environment with a consistent 
message about learning. Another example is an aspect of 
communication ability, writing. relegated to English classes; 
all professors, 'evelop that ability, are aware of writing 
criteria, and as students for application of writing skills in 
their courses, translating the criteria into- the context of a 
^particular discipline (develop a speech on your project for the 
next science convention; write up your results in scientific 
form). Thus, students cannot avoid development of these critical 
abilities. Reinforcement of abilities is pervasive, and faculty 
work to define them, and to identify the unique ways abilities 
are expressed by students, T- is not any particular teaching 
technique (group discussion, internships, exper ience-based 
learning in the classroom, self-assessment using one's videotaped 
speech, skill classes to prepare for college courses) that 
characterize the faculty's overall strategy. Rather, it is the 
systematic, constantly evaluatec use of these «*trategies, in 
relation to v*iat is learned and the student's developmental 
level, that characterizes the faculty's approach.^ 

Assessment is ^ an important part of the Iz-^rning process. 
Teaching strategies incorporate an assessment process that makes 
student/faculty interaction and systematic, detailed feedback an 
expected part of learning. Students are not in the dark about 
how they need to improve, relying only on a "B" or the comment 
"good work" to guide the development of their next proj ct.- 




ERIC 



93 




There is a set for learning «that crosses disciplines, and 
continues after college. This set includes the value of 
continuous self-assessment against the standards of peers, 
faculty, external assessors and the discipline or professional 
area. Since assessment is a process, learning is seen as 
continuous, where perfonnance incorporates not only knowledge, 
but its application. The student experiences this consistent 
message about learning, and about becoming independent and 
autonomous in directing and assessing her own learning. 



Finally, the institution itself is student-focused. Demands 
for the consistent and common reeval uat ion , research and 
redefinition of abilities, their teaching and their assessment 
comes from a systematic attention to student outcomes, and 
student perceptions of the^r learning. New student populations, 
new student problems, new issues for students are the driving 
force of institutional change, coupled with organizational 
mechanisms that ensure that well thought out change occurs, and 
that thuse changes are consistently evaluated. Change is a 
collaborative faculty effort, j.'st as' the development of the 
curriculum and its evaluation is a collaborative effort which 
transcends departments and divisional structures. 



Stud'' X Changes in Performance on 
CoJIege Defined Abilities 

Students Learn Complex Abilities 
in the Curriculum 

Students have consistently shown change on the college's own 
measures designed by the faculty. Each graduate ha ^ , along the 
way, engaged in more than 100 active performance assessments in 
her various courses. Faculty deaign each assessment to elicit a 
particular level of one of eight major abilities, using the 
course's discipline content as a context. Each graduate's 
performances have been v "riously assessed by faculty^ peers, and 
community professionals (and always by herself), according to 
criteria that remain stable across all disciplines. 



We think it is important .that so many students have shown 
consistent change through this complex network of performance 
measures. It suggests that the complex outcomes identified by 
the faculty are indeed developable, and visible in performance to 
both faculty, students, and professionals from outside tiie 
college; that a complex ability is recognizable across settings , 
despite the varied forms it may take in different disciplines and 
professional environments; and that such abilities can be 
d ev e 1 o pe d se quen t i a 1 1 y to increasingly complex levels. 



From a validation point of view, two questions immediately 
arise* First, what broad measures of student performance in the 
curriculum are appropriate indicators of student outcomes, such 
that changes on these measures could be related to changes on 
measures of human potential drawn from outside the college? 
Second, are the college's assessment measures themselves valid? 
And do the faculty indeed share the kind of condensual perception 
of student performances, the inter-rater reliability, so that we 
can be satisfied that the progress students make is actually 
there? 



Consistent with our triangulated validation model, we 
selec ted three broad messur-^s o f student per formance in the 
curriculum: academic reports, external assessments*, and an 
integrated evaluation. 



STUDENT OUTCOMES 



PERFORMANCE 




PERCEPTIONS POTENTIAL 



We validated these measures, along with three other 
facul ty-designec^ external assessments in Communications, Valuing 
and Soc ial Interaction to further test whether ins true ted 
students performed better, whether criteria were valid, and 
whether expert judgment is reliable. 
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Academic Reports 

One aggregate indicator of student performance in the 
learning process is measured by the number of performance 
assessments they have completed at any one point in time. This 
measure allows faculty to obtain a running picture of a student's 
progress. Such a measure reflects the degree to which a student 
has performed the abilities and to what level of complexity. 
Number of credits achieved and semesters attended are also 
recorded. These measures can be used to compare students' 
progress on external measures. Our finding here is that students 
do vary in their rate of learning the abilities, which supports a 
ciirriculum design that^llows for such variability. 

External Assessments: 
Integrated Competence Seminar 

Before our validation research began, the faculty had already 
moved to design certain out~of-class assessments that would 
function as external criterion measures. At the midpoint of her 
college career, for example, faculty required each student to 
participate in a half~day interactive simulation called the 
Integrated Comptence Seminar (ICS), designed to elicit her 
performance of five major abilities at once. The ICS, rated by 
expert judges, is a "content^ fair" cross-disc ipl inary measure of 
the abilities students were developing and demonstrating in their 
several courses (Alverno College Assessment Coimnittee/Of f ic e of 
Research and Evaluation, 1982). Assessor judgments were 
reliable, but the three exercises that make up the simulation 
differed in difficulty and validity. The "In-Basket," a measure 
oi Analysis and Problem-Solving worked best. The measure of. 
Social Interaction, judged from videotaped group xliscussion , did 
not correlate in the expected directions with other measures* 
Social Interaction is an ability that has only recently been 
developed through systematic instruction; perhaps we can't get a 
handle on an adequate measure yet. The importance of this 
ability for future performance at work was underscored both in 
studies of alumnae and professionals. Partly as a result of this 
study, faculty are currently testing out a substantially revised 
ICS, and including demonstration of all eight major abilities. 

In teg rated Eval uat ion : 

Faculty Rating of Student Performance on 
Six Performance Characteristics 

Another such measure is a summary rating faculty give each 
student at the end of each year in college, reflecting their 
judgment of her overall performance on her assessments that year. 
That rating includes six fairly intangible performance 
character ist ics (Awareness , Creativity , Independence , 

Integration., Commitment, and Habituality) which function in 
relation to the more tangible assessed abilities much like 
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adverbs function in relation to a verb. Our five-year study of 
this rating again confirms the high agreement among faculty as 
expert judges, and indicates that it validly measures student 
change in both cross-sectional and longitudinal comparisons 
(Alverno College Assessment Committee/Office of Research and 
Evaluation, 1983). Although we ' have not yet been able to 
validate its power to discriminate among the six -performance 
characteristics, the rating is more highly correlated with 
student performance on cognitive-developmental measures, than 
with those that measure more specific abilities. This suggests 
that faculty ratings on the six performance characteristics are 
tapping underlying patterns of student development. Faculty as a 
whole recognize these student differences in developmental level, 
and we infer that this awareness is reflected in instruction and 
assessment . 

Faculty continue to use the rating extensively, not to 
accredit students but to learn more about the characteristics of 
performance, and to challenge their theory and practice of 
assessment through this shared assessment experience. Currently, 
faculty are reviewing both the definitions of the characteristics 
and the judging process for assigning ratings in preparation for 
developing a better instrument and expand ing its use for 
describing the unique ways students demonstrate their abilities. 

External Assessments : 

Communications, Valuing, Social Interaction 

Faculty have also created generic out- vof-class measures for 
several of the eight major abilities. Each of these elicits the 
student's performance in that ability at the level required of 
all students for graduation (further "advanced" levels are 
required of each student in selected abilities, depending on ^er 
major). So far we have studied three such generic measures, 
those designed for Communications, Valuing, and Serial 
Interaction . 



Our study of the Communications generic instrument indicates 
that it validly discriminates instructed from uninstructed 
performance as does the Valuing generic instrument. Weekday 
students performed better after two years in the learning process 
in speaking, writing, listening and reading criteria than Weekend 
entering students . On level 4 of the Valuing process , Weekday 
students performed better after two years of instruction than did 
Weekend entering students (Friedman, Mentkowski , Earley, Loacker 
^ Diez, 1980). More important, patterns of student performance 
validate the sequential levels of Communications. The cumulative 
sequence of levels 1, 2, 3 and 4 of Communications was confirmed 
for instructed students; v^eekend entering students used a 
different sequence. For Valuing levels 1, 2, 3 and 4, levels 2 
and 3 were found to be similar in complexity for students. For 
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the Social Interaction generic instrument, we again have had more 
difficulty demonstrating that instructed students perform at 
higher levels than uninstructed students. We did find that 
instructed students interpret social interaction skills 
differently from uninstructed students, and maturity and 
motivation affect p>erformance in a group discussion (Friedman, 
Mentkowski, Deutsch, Shovar & Allen, 1982). 

Creating Strategies for 
Evaluating and Revising 
Instruments 

These studies did lead us to try out a variety of strategies 
for validating these nontradit ional assessment techniques. Some 
worked better than others, depending in part on how well the 
ability we are measuring is understood. Abilities like Social 
Interaction are new to higher education instruction, and we have 
a long way to go to adequately validate these kinds of measures. 
We have found that our older college population helps in this 
regard because we get a better picture of just what aspects 
develop through informal learning. And some quantitative 
strategies work better than others in showing differences between 
instructed and uninstructed students. In another series of 
studies with a range of sixteen other measures (Alverno College 
Assessment Committee/Office of Research and Evaluation, 1980), we 
found that criteria evaluation, establishing inter-rater 
reliability of assessor judgments and pre- and post-^instruc tion 
comparisons were three strategies that functioned well and were 
accepted as workable by a range of faculty from different 
disciplines . 

Consequently, three questions are now routinely asked by 
faculty designers of instruments: Are assessor criteria valid? 
Is assessor judgment reliable? Does the instrument measure the 
effects of instruction? Direct involvement of faculty in 
analyzing student performance data and probing validity questions 
continues to generate a broad scope of validity issues. 



In sum, the faculty demonstrate high reliability as expert 
judges of student performance. We will continue to study the 
generic ability instruments, as well as the Integrated Competence 
Seminar and the Six Performance Characteristics Rating, since 
they seem to function effectively as the college's own external 
criterion measures. This is more feasible than attempting 
validation on each of the hundreds of in-class instruments, which 
are frequently revised and which are often short-lived, and ^lay 
offer intervals for student change no longer than a single 
semester. Other examples of milestone measures are narrative 



Expert Judgment is Reliable 
Faculty -Designed Measures Are Valid 
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transcripts, standardized subject area tests, state boards for 
nursing, senior projects and the generic measures per competence 
or discipline. If the faculty's own milestone measures are 
reliable, thev can use them confidently in the ongoing creation 
and redesign of their in-class assessments. 

Student performance of abilities can be validly assessed and 
related to their performance in the curriculum. And we have also 
found that our understanding of an ability develops as we tr^ to 
measure it. But do si.zh changes also show up in stud>^nt 
interviews of their perceptions of learning and on measures, 
designed by theorists who describe human potential in broad 
growth patterns? 

Students Change in Perceptions of Learning 

Do students see themselves as making changes in performance 
that faculty assess and credential in the learning process? Ihe 
major result from this source of data so far is that students do 
show consistent change during college in their perceptions and 
descriptions of learning. They also identify curricular elements 
that promote their learning. Of almost equal importance is the 
finding that students maintain a pattern across all four years of 
justifying learning in terms of its relevance to their career 
expectations. What changes is how they see the nature and role 
of learning within this stable framework; students do come to 
value liberal learning (Much & Mentkowski, 1982). 

We believe educators need to develop a theory of 
ability-based learning that considers how students experience 
their education. How do students make meaning out of their 
learning? What cognitive, behavioral and af fee tive construe t ions 
are agents of transfer of ability-based learning after college? 
How do students see their learning as relevant to their own goals 
and values? What elements of the curriculum affect development 
of abilities and "learning to learn," from the students* point of 
view? 
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STUDENT OUTCOMES 



PERFORMANCE 




PERCEPTIONS POTENTIAL 

We did conduct a par al I el longit ud inal study of student 
attitudes toward the curriculum. Our first effort in this regard 
was to ask faculty to predict student attitudes (Mentkowski, 
1977a, 1977b). In the spring of 1977, we representatively 
sampled and surveyed half the student body on their attitudes 
toward the Frogiam, Assessment, Learning Activities, whether they 
indeed had a negative attitude toward Problems some students had 
identified in the curriculum, attitudes toward Faculty, Support 
Services, toward the Self, and toward Staying in College. 

Concurrently, we asked each facul ty member to complete the 
survey in the way that he or she thought the modal student would 
respond to each item. Thus, faculty were asked to predict how 
students would respond. The following graphic illustrates some 
of the results . 
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FACULTY PERCEPTIONS OF STUDENT ATTITUDES COMPARED 
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In all but one of the instances, there were no statistically 
significant differences between the actual attitudes of the 
students and the faculty expectations. In one case there was a 
<iifference. Faculty underestimated the strength of students' 
positive attitudes toward Learning Activities. In sum, faculty 
perceptions of student attitudes were on target. But we wanted 
to gc^ beyond whether students liked or disliked aspects of the 
curriculum to their underlying perspectives on learning. 



We now turn to the interviews of student perspectives. We 
gathered the student perception data through a confidential, 
opt_. ended interview format ranging up to two hours in length, 
guided by a protocol of questions and probes. The interview 
itself, the protocols, and a method ^or analysis were developed 
as part of this project (Mentkowski & Much, 1980b). Because this 
measure is lengthy and complex, both to administer and to 
analyze, we selected samples for interviewing from both the 
longitudinal and cross-sectional study populations. The results 
here reflect (^j^ 320 interviews from 82 students who provided 
interviews aj^Ebne year intervals at the end of each year in 
college. These students also completed the external measures 
three times during this period, as did another 37 who were 
interviewed as seniors and as two year alumnae. At the end of 
college, students \i\ the longitudinal sample were asked to 
complete a questionnaire on their careering expectations and 
•goals (Mentkowski, Much & Gi encke-Ho 11 , 1983), to measure 
perceptions of careering. 
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PERCEPTIONS: LONGITUDINAL PERSPECTIVES INTERVIEW DESIGN 



ACADEMIC YEAR 
76/77 77/78 78/79 79/80 80/81 



students (n = 20) 

• 

students (n = 62) 



graduates (n = 37) alumnae (n = 32) 



In all, almost 400 interviews were collected and transcribed. 
Interviews from traditional-age students, and those from alumnae, 
have been analyzed so far . 



Students Become Self-Sustaining Learners 



The interview analysis found that learning as described by 
students is a process of experiencing, reflecting, forming new 
concepts, and testing one's judgment and abilities in action. 
Two aspects of learning that students describe seem to be more 
characteristic of a traditional liberal arts education. Students 
are taught to be objective, to stand back and reflect on their 
experience. They are also consistently exposed to opportunities 
to form new concepts, to complete readings and atter.d lectures on 
theory. Students also came to realize that hands o»i experience 
is critical to learning. They also avowed that using new 
knowledge is necessary to really learning it and that one must 
test new found skills. From the student's perspective, these 
elements fit to ler as a process of learning that describes how 
they learn. 
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LEARNING IS: 



a process of experiencing 
reflecting 

forming new concepts 

testing one's judgment and abilities in action 



Andrea describes this process (student names are fictitious): 

By learning something and going and applying it you can find 
out what worked for you, what didn't, what you really don't 
understand, and then you can go back and question. Or in 
learning new theory, you can apply, you can think back to the 
clients you have had or the situaticas you were in and say, 
'If I would have aown that then, it could have helped me a 
lot.' So I think the application and then going back to 
theory and questioning, helps make it more solid in your 
mind, you can understand it better. 



The interview analysis further identified three major 
components that describe the development of this process of 
"learning to learn." They include taking responsibility for 
learning, making relationships among abilities and their use, and 
using different ways of learning. 



STUDENTS DEVELOP "LEARNING TO LEARN 
IN COLLEGE BY 

taking responsibility for learning 

making relationships among abilities 
and their application 

using different ways of learning 



For example, three students, Blair, Gwen and Julia, describe 
the second element, making relationships among abiliti es and 
their use. 



Biair 

Wh en I first came I thought learning would be taking a book, 
reading it, studying it, and doing well on a test. Now I 
realize it's understanding it and putting it in my own words, 
and relating it to other aspects. They want us to relate 
things, to pull everything together and to understand it 



108 118 



more. So now when I go to study, I say, "Well, this isn't 
just chemistry, it's related to everything else." 



Gwen 

Things are pulled together more for you through the 
competences , chingc pull together more than before . You 
might be taking a math class here, s music class there and 
they really have nothing to do with each otner. But if you 
think about it (like you are doing problem solving in math 
and you're doing problem solving in music theory) it's really 
tha same process. You don't really experience that unless 
you can go/to your abilities and see that it really is the 
s>rfte thing. It's more interrelated and you can pull it 
together more for yourself, 

Jul ia 

You have to take these abilities, like valuing, in different 
classes. So I looked at valuing from the philosophical, 
psychological standpoint in a death and dying course. But I 
also looked at valuing in a biochem course, and in an art 
course. So it has caused me to take things and see them from 
many different points of view and that's challenging. To try 
to get values out of a biochem experiment, that's 
challenging. Looking for relationships in a lot of things 
and looking for universality where there seems to be ngne is 
really hard on your head, it really is. 

Students Identified Curricular Elements 
Most Important to Their Learning 

To v*iat curricular elements do students attribute "learning 
to learn?" The next illustration shows the learning outcomes 
linked to elements in the curriculum that students identify as 
causing their develo|xnent of learning to learn skills. 




STUDENT ATTRIBUTED CAUSE 



STUDENT OUTCOME 



instructor attention, empathy 
feedback, self-assessment 

experiential validation 
instructor coaching 
professional application 
integration of abilities 



practice, feedback 
modeling, peer learning 



taking responsibility 
for learning 

making relationships 
among abilities and 
their use 

using different ways 
of learning 



I'm 



One of the more prominent causes gleaned from the interview 
examples is experiential val idat ion , having to apply abilities 
within and across courses, demonstrating them on assessments and 
during internships, using the abilities in multiple settings. 
Roxanne, Lindsey and Alyson describe this process below. 



I know ^ov sure that I learned my anatomy down pat because of 
the chance they gave me to go into the lab and touch the 
stuff, even if it was a rat's corpse or whatever. I got in 
there and experienced it and that^s one part of the learning. 
But when you have this on-the-job xperience, I learned 
because I I ave to use^^h^knowl edge that I have. So when you 
use it th?n you start to internalize it. 



I think I learn better v^e" I'm actually doing something — and 
then getting criticized on it — getting feedback on it. The 
way most of the faculty structure their courses, they give 
you lecture and experience so you can see what you've learned 
in class is really working. The internship program is very 
good for me. You take all that you've learned in your 
college years when you get out there in a job placement, and 
you see that you've really been learning in school because 
you can use it out there. That what you've done did sink in, 
what the teachers have been trying to teach you. And it's 
not just memorizing. It's something you can actually work 
with. It's the experiences they give you that have shown me 
that I've really learned. 



They've challenged me to use all my skills on the spot. I've 
been encouraged here to learn all about the situation before 
making a decision. And so I assess situations more 
thoroughly in uy personal life and at work before I act. 
Just because here they make me so conscious of that sort of 
thing, of observing, of assessing, evaluating the nature of 
something, putting the pieces together. They keep reminding 
me of that when I get in a situation. I can't just put that 
behind me, that's a part of me now. I go around with my eyes 
more wide open, I think. 

Students also consistently broaden the settings in which they 
describe themselves using their abilities. As they progress, 
they cite instances from work, family and other environments as 
often as their in-class assessments. This indicates that they 
have cognit ively made the transfer which they claim to have made 
experientially. 



Roxanne 



Lind sey 



Alyson 
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Other causes included two variables more likely to be found 
at small colleges: instructor attertiop, empathy and instructor 
coaching. But the other student attributed causes — experiential 
validation, feedback, sel f-assasstnent , practice, professional 
application, opportunities to integrate abilities, modeling and 
peer learning — are curricular elements that already are, or could 
be, used at larger colleges and universities. 

Students Came to Value Liberal Leutning 

Do students oriented toward vocation and career develop 
liberal .learning values?- Alverno students do develop values for 
openmindedness , dealing with multiple points of view, and 
appreciation' for th^ arts and humanistic traditions. A large 
share of students come to college today to have a better chance 
at' a job and career, in contrast to values for personal growth or 
learning for its own sake (Astin, 1983), Alverno students are no 
exception. A major result from our s^udy of perceptions is that 
students do maintain a consistent pattern across all four years 
of justifying learning in terms of its relation to their' career 
expectations. Liberal learning values become attached to career 
and professional valCies, which also become elaborated. 

From the outset, students view learning in terms of their 
career, goals . TiTeiu it ional-age students are critical of many 
kinds of learning on this score at first. As they progre'^s, 
however, students consistently develop the ability to assimii.«te 
widely varying courses into their rationales. 

By the second interview, for example, they argue/ that courses 
in ^'non-major" areas and abilities like valuing and aesthetic 
response **help me to stay opfen-minded*' or "give me a broader view 
of things and people." 'Such descriptions are in turn justified 
by expl anations-t know that as a manager 1*11 have to -deal 

with people from all kinds of backgrounds and help them work well 
together," or "When you deal with (nursing) clients, you've got 
to understand their viewpoint and their values and feelings . . . 
that's part of your diagnosis." 

What is significant here is, f irst , the repeated pattern of 
change from skepticism to assertions of value for . "liberal 
education" experiences, on the part - of studeRts who remain 
primarily career- focused . * Second , the pattern includes not 
simply assertions, which might only be environmentaTly acceptable 
noises. Students make relationships between their concepts of 
learning and their learning experiences, and give concrete 
explanations of how they see these kinds of learning as valuable 
to their careers, and to their personal life experiences. 
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Closely allied to this pattern is the consistent importance 
of competence, or demonstrated ability, in the student's ongoing^ 
enterprise of preparing herself for career role performance. The 
fact that the fac\ilty have explicitly identified abilities within 
their disciplinary subject areas, and have linkeH them to career 
role performance, seems clearly to prcvide students with the 
"missing pieces" to link classroom and workplace in their own 
cognitive structure^. 



At least as important, however, is the steady increase in 
stud "its' descriptions of feeling increased mastery, control, and 
certainty. This seems to relate — as they themselves repeatedly 
avow — to their steadily accumulating experience of not only 
identifying but actual ly being ab le to deracmstrate these 
career-valuable nihilities. The areas of Communications and 
Social Interaction are earliest and most frequently cited, 
perhaps since they involve areas that are particularly 
problematic for the young student just entering the college 
environment . 



Students also consistently broaden the settings in which they 
describe themselves using their, abilities. As they progress, 
they cite instances from work, family and other environments as 
often as their in-class assessments. This indicates that 
cognitively they have made the transfer which they claim to have 
made experientially. Through experiential validation of the 
competences, students are able to construct a justification for 
liberal learning in lAiich personal growth and effectiveness 
mediate between educational experience and concepts .of 
professional role performance . 



Experiential validation, described earlier is a key factor 
in the student's justification of liberal learning as relevant 
for her personal life and career. The ability-based curriculum 
has value for them because they find they can immediately apply 
the abilities in other classes, and in their interpersonal 
relat ionships with friends and family , and in part-time or 
full-time work. 




Durjng Coffege 



Libetat Learning 
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Traditional views of college assumed that students r-^roUed 
to develop liberal learning values, which were expected to link 
to career and professional values when students entered the work 
force after graduation. Our results indicate that students come 
to college seeking job and career security. Liberal learning 
values become attached to these early values. Concerns for 
economic security develop into career and professional values, a 
linking that is reinforced and enriched by the new liberal 
learning values. 



Student Changes in Human Potential 

Almost all colleges promise personal growth outcomes and 
expect that college makes a difference in broad abilities, 
lifelong learning and life span development. Studies of college 
outcomes have shown that college as a v*iole causes change (Astin, 
1977; Feldman & Newcomb, 1969; Heath, 1977; Jacob, 1957; Pace, 
1979). Few studies have demonstrated change linked to a 
particular curriculum. Yet it is more and more critical that the 
curriculum rather than the college atmosphere alone be 
responsible for change. Educators are hampered in their efforts 
at curriculum reform if they do not know how the Curriculum is, 
and is not, effective. Also, a Uiger share of younger students 
work, and spend less time in extracurricular activities and/ or do 
not live on campus. Do they achieve personal growth outcomes? 
And does a liberal arts college also committed to izp students' 
professional development, contribute to the development of their 
overall human potential? 

Many colleges enroll large numbers of older students. Does 
the curriculum we offer build on adults' more elaborated 
experiences? Are adult changes merely a function of greatei 
maturity rather than the curriculum? Do younger students develop 
in a college also* focused on the growth of older adults? 
Questions that guided our research are: (1) Do students change 
on instruments drawn from outside the college, tnat measure human 
potential for learning, abilities and life span development? (2) 
Can we attribute change on these measures to student performance 
in the curriculum? What is tht "value added" to student 
development by the learning process? (3) Does the mature adult 
need education or is experience enough? What are the relative 
effects of age and performance in the curriculum on growth? 

Thus, alongside (a) student performance withia the curriculum 
on Alverno-designed ability measures, and (b) the study of 
student perceptions, we also researched (c) student performance 
on twelve measures drawn from outside the college d'^scribing 
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human growth patterns from three separate theoretical frameworks: 
cognitive developmental theory (Kohlberg, 1981b; Loevinger, 1976; 
Perry, 1970; Piaget, 1972; Rest, 1979a) , experient ial learning 
theory and learning styles (Kolb, 1983), and a recent thrust to 
identify and measure generic abilities that link education to 
performance after college (Watson & Glaser, 1964; Winter, 
McClelland & Stewart, 1981)- Because we can relate variations in 
perfomance on Alverno-designed measures to longitudinal change 
on these outside measures of human potential, we can examine 
whether performance in the ability-based curriculum contributes 
to change in human growth. Because research participants range 
in age from 17-55, we can examine the relative effects of age and 
performance in the curriculum on growth as well. 



STUDENT OUTCOMES 



PERFORMANCE 




PERCEPTIONS POTENTIAL 



First, a word about our choice of measures- One problem in 
measuring college outcomes is that most existing measures are 
unrelated to liberal arts goals. It has not proved possible to 
locate measures developed outside the college which are readily 
congruent with all or even most of the complex major abilities 
that educat^-s identify as outcomes of college- The 
preponderance of available measures focus in the cognitive area, 
from broad developmental measures to instruments aimed at 
particular analytic thought processes. We have not been able to 
approach our design ideal of several external measures 
overlapping on faculty-designed abilities or outcomes. For 
example, measures like the Scholastic Aptitude Test (SAT) or 
Graduate Record Exam (GRE) measure content areas, comprehension 
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and ipemory, but do not measure abilities like critical thinking 
or valuing. 



Another problem in measuring college outcomes is that most 
existing instruments also tend to be recognition rather than 
production oriented. Recognition measures tap lower level 
learning outcomes like awareness, but not the ability to generate 
a response in a complex essay or group discussion. Thus, we have 
had to make special efforts to be able to use performance 
oriented instruments in fairly early stages of their development 
(Mentkowski, Moeser & Strait, 1983). While recognition 
instruments are generally more responsive change indicators, and 
more economical to handle, our focus on the complex, active 
abilities and outcomes that colleges intend to foster has given 
us a special commitment to collaborating with colleagues in the 
emerging field of performance assessment. 



Now a word about measuring change, and a short recap on the 
design and data base. Still another problem in measuring college 
outcomes is distinguishing change effects from the effects of 
initial selection, maturation,^ attrition, retesting and societal 
change. To this end, we incorporated both longitudinal and 
cross-sectional designs. 



HUMAN POTENTIAL MEASURES: LONGITUDINAL DESIGN 
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. . ^ — : — > 

ENTRANCE TWO TWO 
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GRADUATES ALUMNAE 



The twelve human potential measures were administered to two 
complete entering classes and one graduating class (altogether 
about 750 students). The entering classes completed the same 
battery two years after entrance, and again two years later near 
graduation. Thus, we have a set of longitudinal results which 
can be dq|g{> le-checked against results from a cross-sectional 
study of 60 graduating seniors compared with entering students 
^o 'ater graduated (controlling for retesting and attrition with 
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initial selection factors such as disposition to change probably 
uncontrolled). Ihe data on 200 of the students who* completed the 
twelve instrtiments on three occasions provides a parallel stream 
of longitudinal information alonjgside these same students ' 
progrGSsive performances on five col lege-designed measures . 
These twelve instruments plus the five measures of student 
performance in the curriculum , yielded 17, 500 responses . This 
extensive data base was gathered to enable large longitudinal 
sample sizes, and to increase educators' confidence in the 
results. 



The design includes two age cohorts (17-19 and 20-55 at 



entrance) to examine the effects 



mat ur at ion , and two 



achievement cohorts (high and low based on number of consecutive 
assessments completed in the learning process) to examine the 
effects of rerformance in the curriculum. Two class cohorts, 
with the second cohort further analyzed for Weekday versus 
Weekend time frames, further enhance representativeness, although 
only further longitudinal cohorts could truly control for 
societal change effects. Ihe time series design holds time 
constant and allows performance in the curriculum to vary, so we 
can attribute change to performance in the curriculum in the 
absence of a control group of students who did not attend 
Alverno. We also control for the following age, background and 
prograr variables as well as pretest scores when we study the 
effects of performance in the learning process. What did we 
find? 



VARIABLES CONTROLLED IN THE LONGITUDINAL STUDY 
OF STUDENT CHANGE 



BACKGROUND 



PROGRAM 



*Age 
Religion 

Mother's Education 

Mother's Occupation 

Father's Education 

Father's Occupation 
*High School Grade Point Average 
*Prior College Experience 
*Maritat Stttus 



Entering Cohort 
Residence 

(Oorm/Commuter) 
Student Status 

(Part time/Full time) 
Major 



PRETEST SCORES 



* Variable related to some scores at entrance to college 
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Generally, Student Change is Related 
to Performance in the Curriculum 



Students c learly show significant developmental changes 
across all three occasions when the battery of twelve measures 
has been administered (Mentkowski & Strait, 1983), Generally, 
the change that occurs can be related to student performance in 
the curriculum. This is the case even wheti we account for change 
due to the pretest scores, age, religion, parent's education and 
occupation, high school grade point average, prior college 
experience, marital status, year of entrance, living at home or 
oa campus, full or part time attendance, and type of major. 



Looking at the resul ts of all the external instruments 
together, we find first, that students appear to change more on 
these external measures in the first two years than in the second 
two years. But the changes in the second interval are more 
directly attributable to the student's successful participation 
in the college's curriculum. This finding suggests that there 
may indeed be a college atmosphere effect, as studies of college 
outcomes have shown. But the curriculum does have a decided, 
added value as weM. 



Older and Younger Students Changed 



Among the other variables that could account for change, the 
age of the student may be particularly significant for educators 
attempting to serve the "new*' student effectively. Older adults 
change because of the curriculum just as younger students do. 
Our noteworthy finding here is that age does indited seem to 
confer some initial advantages as reflected in the 
cognitive-developmental scores of entering students, but not on 
che more specifically focused generic abilities* This suggests 
that educators can rely on age as an indicator of advanced 
ability with respect to broad cognitive patterns but not at the 
more specific level. More important, older students change. 
Their accomplishments reflect more than a paper qualification. 



Students Synthesized Intellectual 
and Interpersonal Abilities 

In looking for interrelationships among the 

cognitive-developmental patterns, learning styles and generic 
abilities we measured, we have found an unanticipated but 
valuable result. When students entered college, and again two 
years later, student's performances on the battery of twelve 
measures tended to statistically cluster around two separate 
developmental factors — one we call logical or analytic thought, 
and the other we call socio-emotional maturity or interpersonal 
ability. But after four years in college, the two clusters had 
merged. This may reflect one of the most desired outcomes of 
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college, namely, that students are integrating their own 
understanding and use of these two kinds of abilities. 

Students Showed More Change on Recognition Measures 
Than on Production Measures 

We expect that measures that call for recognizing concepts 
require a less sophisticated level of understanding than measures 
demanding that a student produce the concept. Recognition 
measures (ranking statements, multiple choice) should show more 
change; production measures (essay, set of stories) should show 
less. Indeed, the recognition measures in general show more 
change across four years than do the production measures, but in 
the second two-year interval, the trends of that pattern began to 
reverse. This supports a recent trend, to develop measures of 
college outcomes which ask students to generate abilities rather 
than to recognizee or comprehend knowledge. After all, these kind 
of production measures are used by faculty to assess advanced 
level work in a student's major. The cognitive-developmental 
measures and the one motivational measure also gave more 
indications of student change than did those focused more 
specifically on particular abilities or processes. Our 
expectation is that the comparatively smaller indications of 
change on production measures will loom larger in relation to 
long term effects concerning careering or future learning. 

Students Changed on Broad Generic Abilities 



These general findings come to life as we examine the 
multiple patterns of student change that emerge from our look at 
students' developing abilities. Combined results from the 
longitudinal and cross-sect ionl studies using McBer' s Cognitive 
Competence Assessment Battery (Winter, McClelland & Stewart, 
1981), and a more traditional critical thinking measure (Watson & 
Glaser, 1964), show chcnges on the broad, generic ability 
measures of critical thinking, achievement and leadership 
motivation, self-definition and personal maturity. Thus, our 
more conservative, variable-controlled comparisons confirmed 
results from a separate seven college study of student change in 
relation to college-promised goals in which we participated as 
"Clare" College (Winter, McClelland & Stewart, 1981). This also 
supports the recent trend in higher education calling for 
production measures of college outcomes which ask students to 
generate essays or respond in simulations rather than to select 
from a list of alternatives. 

The need for more research on college outcomes measures is 
further undt. scored when we compare results on student change 
from this set of generic ability instruments with those from the 
other external measures. There is more change on the 
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cognitive-developmental and learning styl e measures . Also, more 
of the existing change can be directly related to student 
performance in the curriculum. Why thiV is the case is not 
clear. What is clear is that our interpretat ions must rest on 
observations of which instruments show what kind of change in 
relation to instrument purposes and relationships . Colleges are 
in need of valid production measures of b^^oad abilities, and 
studies like this one can contribute to this common purpose. 



Student Learning Styles 
Changed Dramatically 

Our student interviews independently confirmed a recent 
description of the learning process as experiencing, reflecting, 
forming new concepts, and testing one's judgment and abilities in 
action (Doherty, Mentkowski, & Conrad, 1978; Kolb, 1983). 



CONCRETE 




ACTIVE REFLECTIVE 
EXPERIMENTATION OBSERVATION 




CONCEPTUALIZATION 



Kolb derived this experiential learning theory from theories by 
Lewin, Dewey and Piaget and has researched it in part through 
measures of learning style. 



Further evidence for the Alverno ^udent's growing awareness 
of learning processes are the' dr/nnatic changes appearing in 
students* orientations to learning styles, using Kolb's measure. 
At entrance, both younger and older students showed marked 
preference for "concrete experience" over "abstract 
conceptualizing ," and for "reflective observing" as against 
"active experiment ing." 
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In the first two years, they moved rapidly toward a more 
balanced pattern: by the second testing, they had come to rely 
equally on concrete and abstract modes and to show a similar 
flexibility in choosing either reflective or active approaches. 
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Additional analyses revealed that students who showed high 
achievement in the curriculum changed more, and that the 
curriculum still accounted for change where age, pretest scores 
and the other variables were coiitrolled. Overall, achievement in 
the curriculum had mpre of an effect on changes in learning style 
preferences than did age. 



We find that the growth toward balance among learning modes 
occurs for both younger and older students, although there are 
some differences. Age seems to have more of an impact on the 
concrete/abstract dimension; older students are more likely to 
prefer concrete experience as a learning style at entrance to 
college than are younger students. And younger students seem to 
more easily include abstract conceptualizing during the first two 
years of college than older students, perhaps because of their 
more recent high 8C»iool learning experiences. But older students 
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who are high performers m the curriculum continue to increase 
their preference for abstract conceptualizing during the second 
half of college. Age is not correlated with the 
re fl ec t ive/ ac t ive dimension at entrance. But older students 
appear to more easily include active experimenting than younger 
onos, probably because their more immefliate and long tern 
involvement in work and family concerns calls for more active 
trial of tneir ideas and plans (Mentkowski, O'Brien, Cleve & 
Wutzdorff, 1983). 

Students are, after all, expected to become more versatile 
and habitual abstract thinkers, and they s.iould also be actively 
involved in their world as well as reflect on it. We find that 
this balance among learning modes, which appears so d ranat ic al ly 
in two years' time, maintains itself after entry into a 
concentrated, career-oriented major. 

Students Developed Moral Sophistication 

The study of student perceptions indicated more sophisticated 
changes in valuing, also confirmed through faculty experience 
(Earley, Mentkowski & Schafer, 1980). These changes parallel 
similar development in moral reasoning measured by Rest's 
Defining Issues Test. Students became increasingly sophisticated 
in their use of principled reasoning in resolving moral dilemmas. 
Older students showed generally higher scores than younger 
stUfK^nts at entrance to college, but both groups made gains 
.iuring college, with high achievers in the curriculum showing 
more change than low achievers. These curriculum effects were 
maintain* i when age and the other variables were controlled. 

Tliese results were less strongly realized on the production 
measure of moral judgment, Kohlberg's written Moral Judgment 
Interview. Graduating students showed gains over entering 
students in the cros-^-sect ional study, but age was the 
statistically significant covariate. In the longitudinal study, 
change occurred during the second half of college. Results 
suggest that development shows first on recognition measures and 
later and less strongly on the production measures. This 
supports the general finding that change is gradual on production 
measures of life span growth. 

On Loevinger's levels of ego development, c ross- sec t ional 
results showed that students entered college in transition 
between the Conformist and Conscientious levels. Students 
;»raduated at the Consc ient ious level or at the transit ion to the 
Autonomous level. Longitudinal results showed nc> change. 
Students made p.ains in the extent to which they demonstrated 
Piaget's conception of the logical reasoning and analytical 
thinking structures characteristic of adults. These results 
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appeared in the longitudinal study but not in the cross-sectional 
comparison. 



Younger and Older Students 
Changed Their Ways of Thinking 

Themes and Patterns of Change 

A major goal in researching student outcomes was not only to 
pinpoint curriculum effectiveness, but also to better understand 
the nature of the abilities and how they are developed. Studies 
of alumnae and student interviews helped us describe a theory of 
learning that was based more firmly on student and aljmnae 
experience. The study of student performance on outside measures 
of human potential help us examine the nature of change itself, 
its themes and patterns. This is particularly important as we 
embrace the adult learner and begin to develop curricula that 
consider the patterns and pathways of human growth. As stated 
earlier, we found that students change more on these external 
measures in the first two years than in the second two years. 
But the changes in the second interval are more directly 
attributable to the student's successful participation xn the 
college's curriculum. 

Change is Gradual on Production 
Measures of Life-Span Growth 

We also found that recognition measures show more change than 
do production measures. Students were asked to generate or 
produce their own essays or nther self-generated responses on 
measures based on or created by the major developmental theorists 
(Piaget, Kohlberg, Loevinger, Perry). While change occurred, 
both younger and older student groups changed approximately one 
to two stages or positions in four years. We conclude that stage 
theories of adult development are illuminative for understanding 
student perspectives in classroom interaction and on assessments, 
but that these theoretical descriptions alone cannot serve as the 
cornerstone for curriculum development or for asseissment. 
Facul ty~designed abil it y measures are critical for monitoring 
change in ability development on a day to day and week to week 
basis. Clearly, however, a college needs a concentrated effort 
across the curriculum from the faculty, as well as student 
services. If multiple influences on growth are planned for 
across the curriculum, these gradual changes in adult development 
are likely to occur over time. 

Students Changed on the Perry Scheme 
of Intellectual and Ethical Development 

At th'^ same time, it is important to note that mature adults 
make changes because of the curriculum just as younger students 
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do. But we also found that there were differences in when this 
change occurred on some measures. 



Perhaps the cognitive-developmental model most directly 
descriptive of college students and of primary interest here in 
describing non-linear change is Perry's scheme of intellectual 
and ethical development, which is drawn from interview studies of 
Harvard undergraduates. This scheme describes phases through 
which students move as they respond to the diversity and 
ambiguity encountered in college learning. Ihe following 
illustration gives but a brief glimpse of how the student 
conceptualizes learning according to the Perry scheme. Ihe more 
elaborated and complex model in Perry (1970), and the criteria 
explicating each position in Mentkpwski , Moeser and Strait 
(1983), describe a much more elaborated picture of intellectual 
development . 



DUAUISM MULTIPLICITY RELATIVISM > COMMITMEN T 

WHAT TO LEARN HOW TO LEARN HOW TO THINK THINKING 

IN CONTEXT 



Our own intensive study of 3,000 essays from 750 students 
first demanded that we develop a valid method and sets, of 
criteria for using expert judgment (Mentkowski, Moeser & Strait, 
1983) to code essays (elicited by Knefelkamp and Widick*s Neasui j 
of Intellectual Development). We studied three areas of 
development in relation to the Perry scheme: classroom learning, 
decision-making and career. Students wrote an essay in each of 
these areas and it was rated for Perry' s scheme using our tested 
method and criteria. We found that the measure shows definite 
student change in both cross-sectional and longitudinal studies. 



Students show less change in understanding learning processes 
and roles during the first two years of college and more change 
during the last two. Students use more sophisticated modes of 
decision-making after the first two years of college, but after 
the second two years, students show a sharp decrease in level of 
sophistication in decision-making. Probably whea they are 
assessed near graduation, they are making decisions in areas 
related to future issues, and begin by using less complex modes 
of thinking. Change on careering is upward and gradual, although 
this pattern is statistically significant only in the 
cross-sectional study. Ih following illustrations of these 
patterns summarize a combination of results from Mentkowski and 
Strait (1983). . 
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Other patterns emerge when we compare older and younger 
students. Older students have a consistent edge on younger 
students in decision-making and career understanding at entrance. 
And although both groups change, older students maintain their 
increased sop list icat ion. 
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But understanding of classroom learning processes and roles 
is not related to age at entrance to college. Older students are 
starting at the same place as younger students when they ehter. 
But after two years, older students make more immediate progress 
in understanding such concepts as learning in multiple ways, 
learning from peers, and becoming indepenoent in their own 
learning. But younger students do "catch up" during the last two 
years, \^en they make their leap in development. Formal learning 
experiences are necessary for this enhanced understanding of 
classroom learning processes and roles. Change for both older 
and younger students is due in part to performance in the 
curriculum . 



How does high school grade point average, a commonly \ised 
predictor for success in college, relate to these patterns of 
learning? Student change on any of the three areas of 
development is not related to high school grade average when 
students enter college, nor does high school average account for 
change during college. Apparently we are describing different 
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aspects of intellectual development than are measured by more 
traditional indicators of success in college. After all, grade 
point average in high school is a commonly used oredictor for 
grade point average in college. But learning to become a 
learner, as measured by the essay on classroom learning, rated 
relative to the Perry scheme, appears to be tapping quite 
different, more basic structures of thinking. These structures^ 
over long periods of time, do change as the result of college 
learning experiences. Perhaps it is these new understandings 
that account for students' learning to learn, and we need to 
concentrate on this to develop lifelong learners. 

Change is Not Linear; Both Younger and Older 
Students Showed Recycling 

But what explains these variations in growth? Patterns of 
change do not run neatly in parallel. There are increases at 
some intervals as well as some decreases. Our reading of .he 
evidence on this and other measures suggests that development is 
not linear, and that both younger and older students show 
recycling. That is, development proceeds in a gradual upward 
movement, but when an individual enters a new disc ipl ir sett ing 
or life phase, she cycles back to earlier, I'ess sopnist icated 
modes and strategies of thinking. This may explain why we .jee an 
increase in decision-maKi ng ability on the Perry scheme during 
the first t^ao years, and a decrease after th^ second two years, 
when students are general y faced with the morp unfamiliar 
decisions that leaving college brings. . This recycling is 
described in the following illustration. It shows gradual upward 
movement, with recycling. The three points of assessment show 
that we might see increases and decreases on measures, depending 
on when in the cycle we "catch*' stud it thinking. 




This pattern does seem to suggest the kind of complex 
developmental movement noticed by Piaget, in which a learner may 
revert to employing an earlier cognitive strategy when coping 
with new challen:^e8, entering a new phase of growth, or focusing 
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oa a different ability. It will take considerable further study 

before we can say that these results document this phenomenon of 

d ecalage . But the possibility that we might validly record such 

developmentol complexity is a promising one, particul arly because 

our criteria and method enable us to measure the evolution of 

change, as well as stability in cognitive level or position 

(Mentkowski, Moeser & Strait, 1983). 

* # 

Educators cannot assuma that sophisticated levels of student 
thinking in one area are necessarily parallel in an unfamiliar 
area. Thus, even ^experienced adults can expect benefits from 
formal education.^ Ihese findings argue for multiple, systematic 
assessment of abilities across disciplines, to ver^'^y ability 
development and improve prospects for transfer to other settings. 
Indeed, high performers in the curriculum decreased less on 
decision-making as measured by the Perry scheme than low 
achievers performers did the second two years of college. 

Using Abilities is Learned 

Traditional liberal arts curricula have emphasized 
development ^f cognitive abilities over performance. 
Ability-based learning demands using abilities acrosp multiple 
settings. Results ^rou all three data sources — students, alumnae 
'.ad professionals — confirm that the performance of abilities is a 
developmental, learned process that needs systematic teaching anH 
opportunity ti. r practic**. Increasingly complex per forma i 
4evelxip8 in^concert with gradual ly more sophisticated modes . 1 
thought . ' 

Thus, applicatiot. of abilities cannot bt left to chance. 
Performing critical thinking on a term paper needs to be expanded 
to on*-, ae-spot analysis of situations in the practice of one's 
discipline or profession. Student interview discussion of using 
abil it les has led uc to propose an "extension" of the 
descriptions of intellectual growth on the Perry scheme that 
incorporates this performance dimension. From an initial focus 
on **what to perform" ("Tell me \k\at to do."), the student sees 
performance as a process aad concentrates on various ways to do 
something — "how to perfora." Students' next phase is to begin 
"thinking about performing," conceptualizing theories of action, 
whic' they test out in situations. Only then do students become 
capable of "performing in context" — adapting abilities through 
various performance strategies appropriate in and related to the 
setting or particular situation. 
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EXTENSIONS OF THE ?ERRY SCHEME 



DUALISM MULT .-'L'CtTV RELATIVISM COMIVHTV ENT 

WHAT TO LEARN HOW TO LEARN HOW TO THINK THINKING 

IN CONTEXT 

WHAT TO PERFORM HOW TO PERFORM THINKING ABOUT PERFORMING 

PERFORMING tN CONTEXT 

B«ginnirig 

COMMITMENT .^COMMITMENT 



tult^nl and alumnae interview data also confirm that student 
mmitment \ to personal, career and professional values develop 
hroughout college (illustrated by the arrow above). There is 
also an important performance dimension to commitment; students 
consis ^ lykive examples of how they are acting out of their 
persons' ind t>rof essional value systems. Similar to their 
understa .ing of learning, their understanding of their own 
coiamiti -^nt develops throughout college and they show more and 
more sophisticated behaviors that match their increasingly 
internalized goals and values. 
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HOW DO ALVERNO COLLEGE OUTCOMES RELATE 
TO THE WORLr OF WORK? 



We used three approaches to examin the relationship between 
the outcomes shown by college students and the world of work. 
Three independent data sources result: (a) studies of student 
outcomes described in the previous section; (b) alumnae 
perceptions of the abilities involved in the workplace, and of 
the value of learning in their own evolving life goals 
(Mentkowski, Much & Giencke-HoU , 1983); and (c) studies of the 
abilities actually used by professionals in job situations 
(Mentkowski, DeBack, Bishop, Allen & Blanton, 19b0; Mentkowski, 
0*Brien, ^fcEachern & Fowier, 1983). 



The data from alumnae, and other professionals who are not 
Aiverno alumnae, suggest that graduates experience direct 
transferability into the workplace of intellectual and 
interpersonal abilities learned in college, and there are key 
abil ities exercised by effective professionals which are 
statistically related to college learning and conceptually linked 
to the abilities identified and taught by Aiverno faculty. 
Further, alumnae continue as self-sustaining learners, showing 
"learning to learn" skills that enable them to develop and adapt 
abilities, and to achieve job and career satisfactions. 
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Alumnae Realized Career Expectations 



Graduates are highly successful in achieving their inimediate 
career-oriented goals. Of the 197b class, ninety-two percent 
were employed two years after graduation, a percent also achieved 
by the 1983 graduating group a year after college. 

To enable a cross-sectional comparison of graduating senior 
expectations with alumnae realizations, all 63 two-year alumnae 
in the class of 1978 who graduated from the outcome-centered 
curriculum implemented in 1973 were surveyed in Spring, 1980. 
Filty-six responded to the Alverno College Alumna Careering 
Questionnaire (Mentkowski & Bishop, 1980a), providing information 
on work and career history, expectations and satisfactions, 
continued education planned or achieved, and a variety of 
judgments about the value of college preparation for educational, 
personal and career goals. 

A parallel form (Mentkowski & Bishop, 1981) simultaneously 
surveyed 63 of 6b graduating seniors fron the first longitudinal 
cohort for similar information except they were asked to 
anticipate career satisfactions. Thirty-two of these two-year 
alumnae had been interviewed with the Student Perspectives 
Interview (Mentkowski & Much, 1980b) at graduation and were now 
invited for another m-depth two to three hour Alumna 
Perspectives interview (Mentkowski & Much, 1980a), to assess 
their perceptions — and to infer performance. These 32 women also 
completed the careering questionnaire. 



STUDENTS 




PROFESSIONALS 
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Graduating students almost uniformly expect to work after 
college. Ninety-six percent of the alumnae surveyed actually did 
seek employment upon getting their degrees. Of the 92 percent 
who were successful, 89 percent found positions directly related 
to their majors. Since our background data also indicate that 
these women are more likely to obtain professional positions than 
their mothers, education clearly seems to function for them as an 
effective route to professional careers. Graduating seniors had 
higher career expectations than alumnae were able to realize in 
two years, but alumnae rated aspects of satisfaction with their 
f irst positions and potential for advancement as above average . 
Alumnae also show a more positive attitude about their college 
learning after two years than seniors express at the time of 
their graduation, although both groups rated their college 
experience as above average on a majority of items (Mentkowski, 
Much & Giencke-HoU , 1983). 



Both Intellectual and Interpersonal Abilities 
are Critical for Effective Work Performance 

Given their generally positive attitudes toward college 
preparation, how did alumnae abilities carry over to performance? 
Faculty identified intellectual and interpersonal abilities for 
the new curriculum based on experience as educators and 
professionals, and on an analysis of academic disciplines and 
literature reviews. But would these same abilities fom a basis 
for effective performance at work after college? Evidence from 
che two independent data sources shows that both intellectual and 
interpersonal abil ities are critical for effective work 
performance . 



Alumnae Stressed the Importance of Both 
Intellectual and Interpersonal Abilities 

Of the 60 alumnae studied intensively as seniors who 
completed the questionnaire two years after graduation, over half 
were involved in two to three hour semi-structured, confidential 
interviews of their perceptions on work and learning. In 
analyzing the interviews, we found two major categories of 
complex abilities that were equally important in managing their 
work role and careers. Both younger and older women, across all 
professional groups, cited reasoning abilities — using such terms 
as "analysis," "problem solving," "dec ision-making ," "planning" 
and "organi zat ional ab il it ies" — as important to their career 
performance. Alumnae also consistently emphasized interpersonal 
abilities learned in college as critical to effective work. The 
fol lowing are some alumnae examples of us ing intel lectual 
abil ities . 



Clarice 



When 1 think about how I learn now 1 realize that the first 
thing 1 always do is observe a. given situation or problem- 
Then I try to say, "What is the whole and what are the parts, 
what's the relationship." Ihen 1 say to myself, "What do 1 
need to do about it," So then 1 start in my problem solving 
steps. It is more and more becoming a permanent thinking 
mode for me. 1 know what the concepts are. I understand 
them. I know how to apply them. If your plan of action does 
not give you the end result, you can say to yourself, "1 
tried, I tested it and it didii t work. I will no see if 1 
can try something else." It moves you along, you're not 
stuck at dead center. There is something very freeing about 
being able to think like this. 



Megan 

The thing about my job is that it's never the same. Another 
skill I probably should have mentioned, along with 
communicating, is analysis because you do work very 
independently. You have to take the specific individual 
cases, look at them, determine vAiat is needed, what needs to 
be done, get financial statements or whatever from taxpayers, 
analyze them and try to determine a course of action ... 
So that in itself 1 guess is a learning experience because 
every case is different and you do have to use these skills 
over and ovei . 

* 

April 

That year in my classroom, there was something major that 
w-nt on every week. If it wasn't a problem with things 
missing, it was a problem with parents. I would come home at 
the end of each day or week and write down all the things 
that happened — just the bare facts. This is what happened, 
this is what I said, this is what they said. Then 1 would 
reread it and look at it and analyze what the real probler 
was—why things didn't go the way they should have or the way 
I would like them to. 1 would think about what 1 would do if 
a particular situation would come up next, and then 1 took 
action on it . 



The following are some alumnae examples of using interpersonal 
abil ities . 

Rebecca 

In interpersonal situations, in college and different 
sociology courses and the interpersonal skills classes and 
communications, you were given situations and always had time 
to read the materials for that particular course meeting. 
You had time to think of how you were going to handle a 
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situation before you went in and conversed with your 
counterpart. You had anticipations of the outcome and those 
situations base ' on the knowledge that you had and the 
knowledge that your fellow classmates had. It's more 
immediate at worK. 

. . . During this last transition I was being tested. I must 
say, 1 had never ever experienced anything quite like this 
berore .... There was just this whole trial period of 
seeing how far the people on the floor could push me, the 
hourly production workers and the supervisors. Because we're 
monitoring production . . . we have authority and 
responsibility. However, if a production person doesn't want 
to take care of a problem they don't do it. This is where 
this whole business of the interpersonal comes in. I 
remember one day where 1 held some pallets and the crew 
superintendent came up and he started screaming at me. He 
said, "Why are you holding these pallets . . . What is wrong 
with you, there's nothing wrong with this." This big 
harangue must have taken 13 minutes. He took my ticket I had 
written up and ripped it up in front of me. Power play, 
right? Well how do you deal with something like that? So 1 
said something like, "Well ... 1 hope you intend to rewrite 
those hold tickets." After this whole harangue he did 
rewrite those tickets. Later he said, "1 have to 
congratulate you on how you handled yourself ," because 1 
remained cool, I didn't lose my temper, I remained 
reasonable. He said, "The last person I did that to, a woman 
of course, started to cry*" He's constantly doing these 
things to find out how I'm going to respond under pressure. 



Social interaction , communication — not just standing behind 
and watching but opening your mouth when you have 
questions — those are abilities 1 use, definitely. At first 1 
was afraid to adk doctors questions because I thought my 
questions were dumb, but 1 don't feel that anymore. If i 
have a question or if they wrote something 1 don't agree 
with, I'll speak up. 1 guess social interaction is the main 
thing, your communication skills primarily because you're 
more successful with your decision-making. I've been around 
too many nurses who don't open their mouths and I've seen the 
results of that, and people that speak up too much and the 
problems they've had. 



Celeste 
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Practicing Professionals Also Used Both 
Intellectual and Interpersonal Abilities 



Our studies of etfective professional performance were 
designed to build a bridge to professionals who were not Alverno 
alumnae in order to validate abilities the f.aculty had 
identified, and to create learning and assessment tocls based op 
outstanding professionals' job performance. Eighty nurses trom 
three health care settings (community, long-term ca-e, acute 
care) and over 100 women managers and executives from 53 private 
corporations provided us with job performance interviews and 
careering histories. Perceptions were assessed through ratings 
of abilities critical to education, selection and performance. 



Both studies yielded models of broad abil ities that 
characterize effective on-the-job performance and showed a 
remarkable tf^imilarity to those identified by the faculty (Alverno 
College Nursing Faculty, 1979). The following illustration lists 
the nursing (Mentkowski, DeBack, Bishop, Allen & Blanton, 1980) 
and managerial competences (Mentkowski, O'Brien, McEachern & 
Fowler, 19fe3) in the order in which they were performed, from 
most often to least often. 
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Nursing Competences 



Management Competences 



Helping 
Influencing 
Independence 
Coaching 
Conceptualizing 
Ego Strength 
Emotional Stamina 
Reflective Thinking 
Positive Expectations 



Proactivity 

Diagnostic Use of Concepts 

Development of Others 

Accurate Self-Assessment 

Efficiency Orientation 

Expressed Concern With Impact 

Conceptualization 

Self Presentation 

Perceptual Objectivity 

Oral Communication Skills 

Use of Unilateral Power 

Self-Control 

Management of Groups 

Positive Regard 

Use of Soclall:;ed Power 

Logical Thought 

Stamina and Adaptability 

Spontaneity x 

Specialized Knowledge 

Concern With Affiliation 



Intellectual and interpersonal abilities had equal importance 
in the ability models. Effective nurses used "coaching" to 
change clie/rt attitudes and behavior. In "conceptualizing," they 
created patterns of data, identified health problems, and gave 
rationales for treatment plans. Managers were equally likely to 
use intellectual abilities (thinking through problems, applying 
past experiences to interpret events, using a framework to guide 
analysis and actions) as they were to use interpersonal abilities 
(using power, developing subordinates, managing groups). To 
ensure effective career performance for their graduates, colleges 
will have to focus not only on the development of cognitive 
skills, bu^ also their int'^gration with high level interpersonal 
skills. 
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More important, the competence models suggest a sequence in 
the development oi these abilities . For example, "helping'* 
behavior by nurses seemed basic to "influencing" clients to 
change, which formed the foundation for "coaching" clients to 
make their own changes toward better health. For managers, some 
personal maturity and intellectual abilities preceded the 
development of interpersonal and entrepreneurial abilities. This 
suggests that personal growth is a key to development of other 
abilities and that a liberal arts degree can contribute to 
continuing ab it ity development . "Accurate self^c ^sessment" also 
showed up as a critical ability for managers. In the following 
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sequential model of managerial abilities constructed from path 
and correlational analyses (see Mentkowski, DeBack, Bishop, Allen 
& Blanton, 1980, for technical details), "accurate 



self-assessment" is third in the sequence. This is some support 
self-assessment as a prerequisite for other abilities, thus 

a 



for r ^ 

confirming that it be developed in college for use early in 
manager's career. 
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SELF- 
CONTROL 




SPONTANEITY 




ACCURATE 
SELF ASSESSMENT 
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WITH POWER 
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CONCEPTUALIZATION 



PROACTIVITY 



Hypottwtical Mode) of Competanc* m Woown Manager* and Executives 

NOT E Bivanate correlition coef f icienti ire p)aced tn p&rentheses Numbers not m perenthetts are path cotff iciants 



Since both models provide behavioral descriptors, and the 
managerial model is based on another study (Boyatzis, 1982), the 
models can be used to evaluate other programs in higher education 
as well. Because we have collected over 1,000 critical incidents 
of nursing and managerial work performance, W2 have a data base 
for developing mere effective case studies and assessment 
cr iter ia . 



Abilities Function as an Organizing Principle 
for Role Performance and Career Satisfactions 

Abilities Structure Performance at Work 

What are the consequences of organizing learning in terms of 
abilities? AJ tbough alumnae use abilities taught in the 
curriculum, they also use abilities to create a theory of action 
that gets tested out in various work situations. This is in 
sharp contrast to the view that technical knowledge alone is the 
basis for effective performance. Alumnae do say they learn new 
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technical skills, but they do not emphasize this knowledge when 
they describe how they go about deciding what to do ("My job is 
never the same , , . I use communication and analysis because you 
work very independently , • • you have to analyze the financial 
statements from taxpayers and determine a course of action"). 

Analysis of work behavior of experienced, outstanding 
professionals who are not Alverno alumnae shows that 
professionals use the abilities, and that abilities are an 
organizing principle for role performance. In managers* 
performance interviews, we coded few examples of using 
"Specialized Knowledge" as a basis for performance. Managers use 
these skills, but when they describe what they do in situations, 
broad abilities form the basis for their actions. 

Technical Skills Are Not Enough 

The fact that professionals in both studies demonstrated a 
wide range of complex abilities shows that graduates with 
functional or technical skills alone will not be effectively 
prepared to meet the demands of either nursing or management 
positions. Outcomes developed by colleges need to include more 
than the knowledge compon^^.it of abilities. Abilities — which 
cross position levels and even careers — can be abstracted by 
colleges and built into general education curricula. Abilities 
that are profession-specific (e.g., "entrepreneurial abilities" 
for managers or "coaching" for nurses) become the cornerstone tor 
further development in majors. 

Certainly specific training is needed for any entry level 
position, but for persons \Aio plan a career in the two 
professional areas we researched, an education that prepares them 
for the future will include learning to integrate a number of 
abilities, to test them out in a range of actual work situations, 
and to critically appraise one's own performance. 

Alunnnae Experience Competence 
and Career Satisfaction 

Besides the abilities themselves and how they can be used to 
structure work, one of the kinds of learning that becomes most 
critical to career development and career management is the sense 
of competence. The concept of competence is clearly important as 
graduates organize their career role performance around abilities 
and try to improve it. But the experience of competence is a key 
factor in career management and job change. Graduates viewed 
work not just as a job, but ts a career that changes and develops 
through work experience. Jlder women had a specific direction 
toward long range career goals; younger ones were raore 
immediately focused on developing competence in their present 




job. But for all oi them, career satisfaction is strongly 
related to their experi<*nce of competence on the job. 



Experiencing competence seemed to be a critical factor in 
whether she changed her job or career, and seemed to "carry over" 
froin one job to another* While graduates adapted to problems 
encountered in the first two years of work, including the 
traditional "reality shock" or disillusionment experienced by 
most new graduates, the persistent feeling of not being able to 
perform in the job role, for whatever reason, led to change of 
jobs , if not career fields • Such changes were generally 
success ful and appeared to reestabl ish the woman* s feel ing of 
professional competence^ the basis for her self-esteem. 
Apparently, fehese graduates can change jobs and careers 
effectively if they have a sense of competence and strive to 
develop it in whatever role they choose. 

Using abilities led to self'-conf idence on the job and was the 
basis for job satisfaction. Being able to perform abilities led 
to staying on the job. 



But another important indicator of job satisfaction was the 
degree to which alumnae experienced continued learning, an 
intrinsic value which motivates not only career development and 
job choice, but also determines whether an alumna continues to 
develop and adapt her abilities. 



Some abilities are developed more through education than 
experience on the job. Nurses with a bachelor's degree were more 
likely to demonstrate "coaching," an ability, that requires a 
complex form of helping the person to change his or her behavior. 
Nurses with more education were also more likely to demonstrate 
"independence." Those abilities developed through experience on 
the job, like "influencing," should be part of learning 
experiences coordinated with ^ff ^campus york placements . Both 
experience and education wera related to "conceptualizing." More 
experienced nurses showed more "conceptualizing," but nurses with 
a baccalaureate degree made fewer conceptualizing errors that put 
clients at risk. 



USING ABILITIES 




SELF-CONFIDENCE 



CONTINUED LEARNING 



^ JOB SATISFACTION 



Education Develops Some Abilities; 
Experience at Work Develops Others 
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While level of education was less related to performarce for 
women managers, those who had completed a management training 
program showed more "stamina and adaptability" and more "use of 
socialized power" in dealing with subordinates. Those who showed 
more rapid advancement in their company, and most likely a wider 
range of experience, demonstrated more "accurate self-assessment" 
and better developed "self-presentation" skills. 

Both Perceptions and Performance Are Important 

Sources for Validating Outcomes 

In both studies of professionals, we researched abilities 
through performance interviews, and also asked professionals to 
judge a range of performance characteristics. Managers generally 
perform abilities they independently judge as characteristic of 
outstanding performers. We found much less congruence between 
the performance of nurses, and those charac taristics of job 
performance nurses judged as critical for education, selection 
and descriptive of outstanding peers. The findings do allow us 
to identify those abilities that professionals don't demonstrate 
but identify as important — such as negotiating and networking in 
management — that signal abilities that should be part of the 
manager's repetoire. On the other hand, abilities like 
demoiist rating self-control are more important for effective 
performance than the managers realize, and this finding suggests 
that curriculum objectives be tied to the study of actual 
performance. The assessment of competence is important to 
realizing long-te^m goals (Klemp, 1980). Finally, the fact that 
both performance and perceptions are useful and perhaps 
contradictory data sources supports our plan to use performance 
measurer in our future alumnae followup studies. 

Alumnae Continue As 
Self-Sustaining Learners 

Alumnae Continue to Develop and Adapt Abilities 

Abilities are refined depending on their specific 
application. They are also combined in various ways given 
situational demands, and they arc adapted to fit a particular 
action plan. Alumnae spoke again and again of combining ("trying 
to take everything and put it together in a workable process") 
and modifying their abilities ("l should not have come on «o 
directly in getting him to try out this new equipment. 1 needed 
to show him how we can get the work coordinated and out faster, 
first"). In the studies of professionals, effective managers and 
nurses consistently used a combination of abilities in a single 
situation. 
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Alumnae Show Learning co Learn 



But how does this new learning happen? While Argryis and 
Schon (197A) have described the imj)ortance of a theory of action 
in effective performance and McClelland (1975) has demonstrated 
that abilities a^e a basis for effective role performance, 
researchers and educators still mast detrionstrate how this 
happens. Educators question how and why abilities get developed 
so they can foster their in collage. As we analyzed the alumnae 
interviews, a picture of ability-based learning began to emerge 
as it was practiced at work. Several components of "learning to 
learn*' characterized alumnae behavior. 



LEARNING TO LEARN 

Experience learning as a 
continuous process 

Tie knowledge, theory and 
experience to productive action 

Apply abilities in action, get a 
response, and adjust accordingly 

Integrate and adapt abilities 

Think and perform in context 

Alumnae experience learning as a continuou s ^ ^ocess . While 
they are highly successful in achieving their iiamediate career 
goals, they continue to regard learning as a major value and an 
important part of their lives after college* Their work setting 
makes major demands and opportunities for continued learnings an 
important part of developing competence in the job role. At t^^ 
same time, t-hey describe learning as an intrinsically rewarding 
activity which motivates career development, including job 
choice . 

'As educators, we also hope graduates do continue to have 
significant learning experiences after college. Abilities 
learned in college are an important stepping stone to effective 
performance ("What 1 learned most from Alverno and what's helping 
me most in my learning now is the whole orocess of learning, of 
starting and building on what you know, taking it from there.*'). 
Learning experiences are recognized and sought because ''There's 
always a different situation that might come up Another 
graduate comments that) "Alverno taught me that I like to learn 
and that I am capable of learning." 

Graduates consistently speak of the impv rtance of learning as 
"part of life," "part of my job" or "part of the person I am." 
An older graduate in management talks of her career plans in her 
current setting and adds, "If the learning starts tapering off 1 
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would consider s^^^^g to another company, because 1 cannot be 
stagnant m learning.'* A young nurse affirms, "To me, living is 
learning 

Alumnae also seek more formal learning. In assessing 
commitment to more formal learning by graduating seniors, 36 
percent expressed definite plans to continue their education 
after college, while another 57 percent indicated a desire to 
continue but did not specify a time line. These expectations 
were more than realized by the group of alumnae two years out of 
school. Forty-one percent dia complete additional education, and 
of the 5^ percent who did not, the majority expressed, plans to 
acquire Additional formal education at a future date, showing a 
high commitment to continued learning. 

Another element of learning to learn is to tie knowledge , 

t ' pory and earperLence to productive action . Alumnae describe 
nutting these elements together in new situations. One alunna 
comments that, '*When you're faced with a situati.. all of a 
sudden you start bringing out ail tha. you were tau^nt, trying to 
take every bit of education, suggestions from your supervising 
teachers, and pul it together." Another says, "i have been 
gradually learning how to use my small group theorie«i and relate 
them to the strategies others use. I*ve learned how to pick up 
on signals and back off a little bit and develop arother round 
about route." This example raises another element of "learning 
to learn, * that is, a^lyin^ abilities in action , getting a 
r esponse , and adjusting one ' s per formance o r ideas accordingly . 
"I h^d to spend a great deal of time observing, thinking 
retrenching, thai sort of thing," says one alumna. Anothe- 
admits, "When I come across a fresh situation I feel i need more 
information so that I respond differently, I remember what I 
read, I sit back and analyze what happened, what I think shouH 
happen ." 

Alumnae also in teg r at e a n d a dapt abil itie s based on this 
experience of "observing, thinking, retrenching." Abilities are 
often integrated and overlapping in practical role performance. 
They range from simply becoming familiar with a new environment 
and new task*^ ("Learning is a big part of what I do because wi t 
I started this job everyt ,ing was new to me") to becoming an 
active learner in trying to carry oul the role effectively (* I'm 
still learning what to use and I'm trying new things every year 
to see what's going to work"), seeking information from others 
arid the situation. 



ideals for performance and accomplishment must be modified in 
the work setting. Because of the environmental press abilities 
iearn'^d in college must be adapted ("I learned you have to 
introduce change. ve^-y slowly and graduf»l ly and to teach about 
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the change before it happens.'*). New atciLudes, beliefs, 
perspectives and elements of self-concept or profeosion&l 
identity are acquire as well k"hy ability to compromise and be 
more tolerant and ^pen-minded hao increased/*)* A young nurse 
who has already made a job change summarizes the processes of 
adapting abilities to new environments ("You have to think more 
in terms ot *riow do 1 do this in this situation?' You are more 
consciously involved in what you are doing,"). 

Central to learning to learn is to think a nd p erform in 
context . We see not the simplistic use of abilities as they have 
been first practiced, but ra :her a sensitivity to their 
discretionary use, depending on the constraints aitd challenges of 
a particular work setting or event. Alumnae see their actions in 
relation to their reading of the situation, and the consequences 
1 ike ly to occur • 



For both nurses and managers, performance of abilities is 
influenced by the context in which it occurs. Of the nine 
abilities identified, nuises in the community health setting 
performed significantly more of them did those in the acute care 
and lo'^2~'-^'""* care settings. The hospital and nursing home are 
generally seen ac more structured and less open to the nurse's 
opportunity to perform the full range of her abilities. Managers 
from larger organizations demonstrate more of the competences, 
like "development of others," "managemr*nt of gro'^ps" and 
"diagnostic use of concept " Larger rgaaizations and some 
types of industries set . to provide mere opportunity for 
performing a wider range of abilities in the managerial role. 
Such influence by the work environment suggests that integrating 
and adaptiag abilities is critical for adequate performance and 
that graduates need to be able to think and perform with 
attention to the demands of particular settings. Context is an 
important factor in work performance. Abilities taught in 
college need to be practiced across settings. 

In sum, college learning and abilities fotrm a foundation for 
role per formaiicf^ after college, but learning to learn is a 
prerequisite to ada, ♦'ing abilities in the role one has. Thus, 
learning* to learn Xb a process that enables adaptability to 
multiple settings; it link.-, abil ity~based learning and work after 
college. 



LEARNING TO LEARN LINKS EDUCATION AND WORK 
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WHAT HAVE WE LEARNED? 



We believe we can show that complex abilities can be 
successfully integrated as a result of a college experience 
stressing an ab il ity~based , outcome~cent ered approach to 
learning. These abilities can be identified, developed and used 
to achieve success in educational and work environments. We can 
also identify contributors to effective outcome-centered 
learnings These include instruction itself, as measured by the 
amount of student change on faculty-designed instruments and 
those d^awn from outside the college, curricular elements 
identif i by students as important for their learning, and the 
impact education on the demonstrated abilities of alumnae and 
other . .0 fessional s . 



Outconnes Are Complex. Holistic Human Abilities 

We researched outcomes through several different frameworks 
and measures. It is clear that definition and measurement of 
college outcomes needs to include a range of dimensions: 
cognitive/ intellectu&l process , af f ective/ soc io-emotional 

process, perceptions, motivation and performance. The outcomes 
studied by our battery of twelve external measures were 
differentiated info two separate factors at entrance to college 
and two years later, but were integrated by graduation. This 
suggests that educators may nead to differentiate cognitive and 
..ffective aspects of abilities in order to teach for them. Most 
educators are struck by the difficulty of any a Lt«itpt-<D separate 
these aspects. Yet attention to each dimension in tur&\may be 
necessary to enable students to integrate tltem later on. wXneed 
to study carefully just how this integration occurs, and \hat 
aspects of the learning [vocess seem to develop this merging. 



It is also clear from our study of student performange on 
external measures that educators defining competences, or 
abilities need to attend to individual differences in levAl of 
cognitive development and what implicaticu^ this has \ f or 
developing instruction. Faculty who rated students on a seQ of 
performance characteristics seemed to be tapping stuqent 
development, since the rating? correlated with 

cognitive-developmental level. If faculty are thus aware of 
students* developmental level — along with the more tangible 
abilities involved in their day-to-day assessments — we might 
inter that they also take developmental level into account in 
instructional planning and their interactions with students, to 
challenge and support student learning. 



Further, complex outcomes need to include a performance 
dimension. Learning to p rform, to link knowledge to 
performance, is a dimension of abilities separate frora the 
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cognitive-intellectual one. This is borne out by the fact that 
tognitive-developmental level correlates with the concrete 
experience/abstract conceptualization dimension oi learning 
styles. Cognitive-developmental level does not jcorrelate with 
the reflective observation/ active experimentation dimension. 
Educators have long sought an adequate learning theory that 
incorporates not only knowledge and cognitive/ intel lectual 
processes, but also the more practical learning that occurs when 
ideas are tested out in actual situations. This practical 
learning can be expected to transfer across contexts to the world 
of work. For students in our study, learning to perform, to link 
knowledge to performance, enabled them to imd reasons for 
learning in a variety of ways. They tried out the competences 
through application to professional performance and in their 
personal life. By doing so, they experiential ly validated the 
competences or abilities they were developing. The concept of 
"competence," which implies knowledge and action, becomes^ ' a 
motivational link as well. Students began to see themselves as 
competent. Thus, outcomes have a perceptual and motivational 
dimension that assist in their internalization and transfer. 
Values and motivation for performance have their roots in 
students' justification for learning as a stepping stone to a 
career and economic mobility. Along the way, self-sustained 
learning, a liberal arts value, becomes part of the student's 
reasons for continuing in college. The student perceives herself 
as a self-directed learner, who seeks "we 1 1-roundedness as well 
as career goals. 

Complex abilities, which include cognitive, affective, 
behavioral, motivational and perceptual components, do fit 
together and/or integrate to some degree by graduation. ais 
suggests that the abil-** ies are holistic , that is, that they 
involve the whole person. 



In this study, complex outcomes or abilities change over 
time, and are related to performance in the learning process. 
Thus, they are developmental or teachable. We can. link outcomes 
specifically to college instruction in five ways: (1) by showing 
that students change on faculty-designed assessments as the 
result of instruction, (2) by the analysis of confidential 
interview^ where s*"udents and alumnae attribute changes in their 
learning to curricular elements, (3) by the analysis of student 
change on the twelve external instruments drawn from outside the 
c lege, (4) by analysis of alumnae ratings and confidential 
interviews where graduates describe using abilities in 
post-college settings, and (5) by showing the impact of education 
on the demonstrated abilities of professionals who are not 
Alverno alumnae. All of these sources validate the testimony of 



Outcomes Are Developmental and Taachable 



Outcomes Develop as the Result of the Curriculum 
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faculty who think they see studeuts learning, external assessors 
who credential some oi these abilities, and the testimony ot 
other students and alumnae who say they are learning them and 
whose reports become more complex in describing their abilities 
in college, at work and in their personal lives. 



Just as student perceptions change, faculty ratings ot 
performance characteristics alsc snow students changing. Student 
performance of f acul ty~def ined abilities of Communications and 
Valuing are related to instruction. Students^ by their own 
report, find Communications and Social Interaction abilities 
useful for functioning in personal and professional roles. 



On the other hand, there are other complex outcomes and 
competences where the link to performance in the learning process 
was less clear (e.g., Social Interaction and other, externally 
assessed generic abilities). Older and younger students differ 
on some abilities and not on others at college entrance, and show 
some different patterns in de>/eloping them. how education 
interacts with experience to enable the student to build on 
informal learning outcomes is important in designing instruction 
to fit the adult learner. These results show that most of the 
outcomes or abilities we studied are developmental and teachable, 
and that we need to more carefully probe the dimensions of those 
other abilities that are less easily linked to college learning, 
college learning. 



Outcomes Develop at Different Times 



Equally important is the time frame for development. There 
are differences in when these abilities develop during the 
college years. And as educators have always suspected, there is 
a difference between the general education experience and the 
later years when the student focuses on a major. Older and 
younger students perform differently with respect to some 
cognitive-developmental patterns and abilities but not others. 
Further, abilities that may be differentiated during the first 
two years of college become integrated during the last two years, 
although how this happens is not clear to us now. The competence 
models developed from effective professionals in nursing and 
management show that abilities differ in complexity and sequence 
and suggest that the pedagogical order of abilities is important 
and can be identified. Professionals more likely develop some 
abilities exclusively on the job. Experience may add to a 
student's ability to take advantage of college, but some key 
abilities critical for effective work performance are clearly 
developed over time m long term formal learning experiences. 
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Developmental Patterns Are Subtle and Complex 



When we look at the rate and quality of change, we note the 
kinds of subtle and complex developmental patterns that. will 
ultimately be of most use to practitioners and to theorists. As 
we study students' developing perceptions, we see that students 
do value open-minded ness and sel f-direc tion and seem to 
demonstrate it increasingly. We would like to be able to account 
for how students actually do undergo the changes that they 
demonstrate in their interviews. It is our belief that 
communicat ing these subtle and complex pat terns across 
disciplines, and to a wide range of faculty, is a prelude to 
identifying the criteria for assessment of these abilities. We 
also think that being better able to define criteria for 
assessment will lead to improved instruction, and consequently, 
improved educational validity of the learning process. Such 
efforts will begin to link developmental theory and educational 
practice (Attin, 1983). 



Students progress tnrough the assessment process with no 
noticeable deficit for the older student. The very structure of 
Weekend College (an alternate tioe frame which requires more 
independent learning) attended mostly by adult students presumes 
that the older adult can movf^ at a more intense rate. Not only 
must she evidence this cognitively, but also in organizing 
multiple roles and responsibilities. The older student's life 
experience is not ignored, and tnere is no evidence of older 
students having any disadvanuage from bning away from formal 
academic work. In fact, there is a cognitive advantage that 
allows them not only to cope with the program but to cope with a 
concentrated pvogram that's even more demanding . Tnat 
presumption is borne out by students attending Weekend College 
who also performed on external iniJtrument s ; changes are not 
differentially less than the performance of students in the 
regular college time frame. And older students also develop 
their abilities as the result of instruction, however, our 
analyses of the external instruments show that age is an 
advantage in some areas initially, because the experience that it 
implies enables the student to take on this cognitive overload 
and deal with it successfully. And by the time they are more 
than two years along, the eaucational environment itself is a 
more likely determinant of learning. At the same time, older 
students begin at the same place as traditional age students in 
some areas >^ such as their understanding of classroom learning 
processes and roles. 

This difference between the older and younger adult shows up 
alter college. The experienced adult has more specific direction 
toward long-term career goals, in contrast to the younger 



Older Students Also Develop Outcomes 
as the Result of Instruction 
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graduate who is more focused on immediate ones. Both groups, 
however, report having to apply the same kinds of abilities to 
post-college work settings — interpersonal and reasoning 
abilities. Thus, adulr.s have an advantage in some areas but not 
in others. But it appears that the ability-based curriculum at 
least is capable of capitalizing on the differential abilities of 
the new learner. 



Another reason for this confidence on our part is that the 
college we have been studying has a traditional mission to serve 
working class students who are often first generation college 
students. For this student group, higher education can build on 
their particular strengths and background and enable them to 
continue capitalizing on it even when they've graduated. Women 
alumnae we studied show upward job mobility compared to their 
mothers . 

Outcomes Include Self-Sustained Learning 
that Links Education and Work 

Career-Oriented Students Develop 
Liberal Learning Values 

Ihis study also shows that the new student can be served when 
educators act on the assumption that abil ities le:'rned interact 
with the student's vilue for learning. Values for taxation in 
this student group are linked to career outcomes. Indeed, a 
career-oriented rationale for college learning seems to describe 
today's student (Astin, 1982). The concern is that work-related 
rationales will bypass or shut out the traditional liberal arts 
values for lifelong, continued learning, for dealing with 
multiple perspectives, for appreciation of the arts and 
humanities, for personal growth, and for benefiting others in 
society. The results from this study indicate that those fears 
are unfounded for this student group. During college, values for 
personal growth and concinued, lifelong learning emerge. These 
values become linked to professional role performance, and to a 
perception of the self as a competent, self-directed learner. 
That these values for learning continue beyond college is 
evidenced by job changes made by alumnae who have inadequate 
opportunities for new learning at work. 

Liberal arts educators do, however, need to be aware of the 
values for learning of its student groups. Apparently the 
concept of competence and learning to learn and to perform are 
strategies that can build a bridge from the practical values 
stude ts bring with them to che more intangible values for 
lifelong, continued learning and professional role 
performance — where benefiting others is a primary aim. (leariy, 
responding to new students means being as insightful as possible 



1^156 



about the reality of the learning experience for them and how 
they relate values for learning to their own goals for being in 
college. 



Learning to Learn Skills 
Are Developed in College 



From the beginning, we were aware that one intangible goal of 
the curriculum was that students develop as independent learners 
able to learn on their own. But we were unprepared for the 
sharply focused emphasis on learning to learn by studen-s, their 
easy identification of its several aspects and their attribution 
to curricul ar elements , the dramatic results in changes in 
learning style, or that both older and younger students needed to 
"learn how to learn" in relation to classroom learning processes 
and roles. Further, this internalized process was a key link 
between college-learned abilities and performing them at work 
after college, and even accounted for job satisfaction. 

We also found that any concept of learning needs to consider 
not only the intellectual part of 1 earning but al so its 
performance dimension. Taking responsib ility for her own 
learning was a recurrent theme in the student interviews, and she 
linked it directly to opportunities for experientially validating 
her abilities. Applying knowledge and abilities, adapting them 
in various contexts and using them in a d iscretionary way 
depending on the situation was described by students and further 
elaborated by alumnae. 

We can now b^^tter describe what it means for a student to be 
able to learn on ner own. And development of these skills is 
apparently an important indicator of her ability to gain from 
college. Some students did not develop learning to learn as well 
as others, and they did not seem to gain a6 much from college as 
a result. We need to !:urther explore this concept, primarily by 
looking at intra-individual differences in patterns of 
development on the several meai>ures : interview, learning style 
inventory and the Perry scheme of intellectual and ethical 
development. Our use of the learning style inventory as a 
diagnostic measure for students in a beginning seminar (Deutsch & 
Guinn, Note 1), our use of various learning strategies ueveloped 
from the Perry scheme in a communications seminar (Loacker & 
Cromwell, Note 4), yields a further test of our assumptions. Our 
efforts to assess experiential learning and to develop a 
production instrument are providing new insights on how learning 
to learn is developed in internships (Mentkowski, O'Brien, Cleve 
& Wutzdorff, 1983). Further, the faculty perspective on how 
learning to learn develops is described in their integrated 
evaluation of student performance (Six Performance 
Characteristics Rating). Results from all these measures will 
help us to develop a diagnostic tool for both faculty and 
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students to assess students' developing ability to become 
self-sustaining learners. Faculty could also then get a better 
handle on who is changing in what kinds of ways, and be able to 
read the signals for change and transition. In doing so, we are 
moving towards the study of more individual patterns of growth. 
Clearly, the development of learning to learn skills and ttieir 
use, as well as the accompanying values aad attitudes about 
learning cannot be left to chance development in a college 
curriculum. 



Learning to learn means discovering how to derive from an 
environment and experience what one needs to know to adapt one's 
abilities. Our results strongly suggest that '.sing these skills 
and the consequent adaptation of abilities is such an important » 
process for the graduate that college learning needs to 
specifically train for it. While college graduates will always 
face disillusionment and the conflict between realizing their 
ideals and making a living, how they deal with the challenge 
seems an important component to making "he transition from 
college to work. Student values for learning to learn are 
realized in alumna motivation to use and adapt abilities to a 
range of contexts. Alumnae test out new ways of doing things to 
find out what will work. Learning how to adapt abilities 
involves a proc:^ss of applying judgment and abilities in action, 
getting feedback and adjusting accordingly. 



That the value for learning is internalized is shown not only 
because career satisfaction is built partly on opportunities for 
new learning but also because graduates go on to more schooling. 
Alverno students come to college for job preparation. From their 
point of view, the college prepares them adequately, and they are 
almost all successful in finding the job they v/ant after 
graduation. Over 40 percent of the alumnae we studied have 
continued formal learning two years past college, and another 50 
perceiit expect to do so in the future. 



The outcomes ot college are generic, that is, they transfer 
to post-college settings. While level of education is linked to 
e f tec t ive per formance on the j ob , t he abil it ies ident i f ied as 
crucial to effective performance, like reasoning abilities, on 
the fa"e of it, are similar ta those identified by educators. In 



Alunnnae Use Learning to Learn Skills 



Learning Continues After College 



College Outcomes Promote Careering 
and Professional Performance 



Abilities Identified by Alverno Educators 
are Demonstrated by Professionals 
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contrast, interper."?onal abilities, long an expected result from 
informal learning alor*^: are critical to effective performance as 
evidenced in the professional's competences and the observations 
of working alumnae. This validates the Alverno faculty's choice 
of abilities like valuing and social interaction as similar in 
importance to intellectual abilities. The finding also suggests 
that all highly complex cognitive abilities be integrated with 
high level interactive ones. 

Clusterb of abilities carry forward from college to tne world 
of work. Whil> they must be integrated ard adapted to the work 
place, they contribute to effective performance. Both competence 
models of professional performance show that professionals 
demonstrate a wide range of complex abilities. While the type of 
organization in which they are employed seems to influence the 
competences tney perform, there are abilities that are generic, 
that transfer across setting and occupation. 

Abilities Learned in College 
Are Used by Alumnae 

There is a remarkable congruence between the abilities 
graduates say they use in the work place and those Alverno 
educators consider important outcomes of college. Intellectual 
and interpersonal abilities are both mentioned as necessary for 
coping successfully with a range of situations. And abilities 
function as an organizing principle for role performance and 
careering . 

Professionals' oerceptions of abilities descriptive of 
outstanding performers were congruent with demonstrated abilities 
on the job for managers but not for nurses. Why this is the c, se 
is not clear, but it suggests our plan to assess the effective 
work performance of alumnae in addition to their perceptions is 
wise. The fact that alumnae have focused on developing abilities 
during college might make for more congruence, but this cannot be 
taken for granted. Since career variables like salary and status 
are not linked to effective performance for women in an emerging 
field like management, colleges seeking to validate their 
curriculum for women alumnae need to rely on performance as their 
indicator in addition to measures of perceptions and self-report 
indicators of career advancement. 

Competence is a Concept 
and an Exp>erience 

College outcomes and work are related very strongly by the 
notion of self-perceived competence. It is a cognitive organizer 
tor learning both in college and at work. It is one of the most 
powerful experiential triggers for development according to 
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alumnae testimony about how they manage their career changes, and 
career satisfactions. Ihe mere act of identifying outcomes and 
giving people a chance to practice them has a powerful impact 
that carries from college to later life. Educators attempting 
outcome-centered education are in large part responding to the 
press for work usable education without sacrificing what 
education has trau itiona] ly meant. It seems fairly possible to 
take a liberal education and define it in terms ol outcomes and 
make taose outcomes experienceable to students, creating an 
effective, lasting link between education and tne world of work 
without having cO sacrifice the value of those complex outcomes 
that motivate the liberal educator. The outcomes of liberal 
education can be identified, and when identified and experienced 
by the student, do persist. 



Vaiidating College Outcomes is Feasible 

Throughout our research and evaluation efforts, we have paid 
particular attention to discussing methodological issues usually 
generated by large scale validation efforts. Several 
methodological contributions are discussed here because they 
point to the feasibility of either starting or continuing such 
work. 

Defining Validity as a Concept 
for Higher Education 

The press for accountability in colleges creates a need for 
evaluation and validation strategies. But assuming that such 
strategies can be applied without concern for the history and 
traditions of the liberal arts would only serve to alienate many 
faculty and administrators. Demanding that colleges now devote 
signiticant effort and resources to establ ishi^ng validity in ways 
outlined by the behavioral sciences alone, ensures the failure of 
the approach. 

Some liberal arts colleges have valid concerns about the 
press for accountability, and reject the assumption that 
establishing evaluation and validity are the most cogent response 
to such pressures. Fears that such attention to evaluation would 
mechanize, or otherwise destroy the primary values of the liberal 
arts have been expressed. If we attempt to identify ana validate 
complex abilities, won*t we end up with abilities that are 
mechanistic and so specific that a quick study can demonstrate 
them easily? Strategies seen as advances in the behavioral 
sciences and educat ional research communi t y may not be 
appropriate for liberal arts contexts. And the practice of 
program (^valuation and validation of developmental outcomes needs 
new pproaches. In any science, behavioral or otherwise, new 
parad ^ras are critical for solving new problems. 
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Tnroughout our reports, we speak to the issue of adapting 
validation goals, strategies and designs to the particular goals, 
strategies and curricular plans of the faculty and involving 
faculty in carrying out validation efforts. Establishing 
validity means first identifying its meaning and use in a 
particular context. We do not suggest that a college incorporate 
our design or methods, but we hope colleges will find some ot tne 
strategies we used helpful to ensure that their definition of 
validity and validation design builds on and is consistent with 
existing academic administrative structures and college-wide 
goals. Wc found validity best defined as a process that is 
developmental, illuminative and diagnostic, and relates theory to 
practice and research to evaluation . Such a definition of 
validity suggests a validation design fitted to the context in 
which ic is applied. The extent to vhich results from validation 
studies can be incorporated into an ongoing curriculum and used 
by faculty to improve it is the ultimate test of their validity. 
In this liberal art^ setting, faculty involvement was essential 
to meeting research objectives. 



Designing Validation Models 



Although we claim that validation models must be designed to 
correspond to the goals and curriculum of a faculty, validation 
models in a particular setting also need to be designed in view 
of the broader issues of teaching aiid learning that challenge 
education today. Findings must be general izable to other 
settings and populations if our concentrated efforts across 
colleges are to lead to exchange among institutions and 
accountability to society at large. We thus designed a 
validation model with four levels of triangulation, building 
externality into every level of the triangulated model so that 
findings could be used not only to improve a particular 
curriculum, but also to generate findings that wouJ generalize 
to teaching and learning settings and populati s At other 
institutions. As we widen the lens, bringing in other 
perspectives and frameworks, we generate results that are of 
interest not only in improving one curriculum, but that have the 
potential to improve education in other settings. This attention 
to external it y al so solves problems of subjectivity where a 
college takes on its own validation since, in this case, student 
and alumnae outcomes are compared to the standards and norms of 
other groups and other frameworks of learning and development. 



There are other inherent contrad ic tions in designing 
validation models than the one just mentioned. Since findings 
trom validation efforts should result in curriculum improvement, 
validation models need to have a dynamic quality that fits an 
ever-cnanging curriculum. Experimental/control group comparisons 
are oUen not possible in settings where variables, treatment and 
setting are changing. Yet the model must allow examination of 
the relationship between student outcomes and performance in the 



curriculum and sUow that instruction enhances student 
performance. Further, models must generate group results tnat 
can influence broader curriculum and program development and 
snape the long-range policies of a college, yet they must also 
generate findings at the more micro level of analysis to enable 
exploration of individual differences and teaching strategies 
that work for individualized instruction. In addition, modt^ls 
must allow a series of findings from a range of data sources to 
emerge over time since a faculty need fast results to solve 
curriculum problems that need immediate resolution. Yet the need 
for long-range longitudinal designs and the complex effort over 
time needed to research some questions means waiting for results. 

The validation models with four levels ot triangul ation also 
works to resolve these contradictions. We studied abilities from 
multiple points of view, across multiple points in time, using 
multiple groups, with multiple opportunities for critique and 
coraparison. This strategy of investigation is consistent with, 
multiplicity in ways of knowing most representative of 
methodology across the liberal arts disciplines. ^-i»i?e-widen the 
lens, we bring more comparisons into play to test out 
assumj>t ions . Longitudinal and cross-sectional designs used 
within the larger model enable us to pinpoint change ad relate 
it to student performance in the curriculum from several 
perspectives. Further, the triangulated model allowed for 
comparisons across various grojps. Longitudinal designs allow 
for exploring unique patterns ixi intra-ind iv idaal change, and 
delving more deeply into single abilities and intra-ind ividuai 
change patterns. -Since the model calls for multiplicity of 
comparison over time, the model generates results in some form 
throughout its existance. Although the longitudinal designs 
employed do not produce immediate results, finding from the other 
designs are generated in a more timely fashion. 

Can a Liberal Arts CoMege Accomplish Its Own 
Evaluation and Validation? 

Dun ig the past decade, responding to demands for 
accountc b il it y usually meant contracting with an outside 
evaluat .on consultant or agency who then developed and executed a 
design Resources for such external evaluations are dwindling. 
And persons doing the research are absent when the real work of 
evaluation begins — implementing the results. We built - our own 
internal and external eval uat ion/ val id at ion mechanism, and then 
monies channelled into validation served as seed money to develop 
the abilities of college staff. While the evaluation staff 
fluctuates depending on availability of outside funds, there are 
consistent, evaluation and validation studies constrntly ongoing 
and vSUpporled by the col le^',e . 
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Ongoing, intra-mstitutional evaluation has raised the 
quality of other internal evaluation conducted i*j che college. 
Faculty outside the behavioral science^ are more willing to 
consider evaluation as part of curriculum development because 
they no longer bear all the responsibility tor a task that has 
its roots primarily outside their ^'^.eid. An fxpert staff is also 
available co assist faculty with grants calling for evaluation, 
ana lacu) ty are more willing to enter inc6 relationsnips with 
outside funding sources. Funding agencies are more likely to 
provide funds for a project that has tt\e mindset and demonstrated 
expertise tor rigorous evaluation and validation. Responsibility 
for self-evaluation encourages close attention to ixercising 
objective , analytical judgment , and to submit t ing plans and 
reports to outsiders for critique and review. 

Developing Participant Involvement Strategies 

One outcome off our e f f ort s was d evei oping strategies to 
involve students, llumnae, and professionals from the business 
and professional community. Methods of assessment often involve 
using an unobtrusive^^easui.e where the participrnt is not sure 
what is being assessed. In our case, we needed to use measures 
that had face validity for a range of persons. We needed to 
generally inform them about our objectives so they would continue 
to participate in longitudinal research. But how could we 
guarantee our results would not just reflect a halo effect or tne 
willingness of participants to "help" us by using as much of the 
acceptable jargon as possible? 



We did inform participants of the nature of our validation 
goals. But we also used a range of complex indicators and 
assessment s ( in-depth interviews ; cognitive-developmental 
instruments which assess growth over long periods of time) to 
help ensure that outcomes were actually there. And we had to 
develop some data a\alysis methods to differentiate beginning 
from fuller understanding of the concepts we were assessing. At 
the same time, we met research standards for objective data 
collection and analysis. 



Using informed participants also served to meet other college 
goals. Creating relationships with alumnae, building bridges to 
the professional and business community, and valuing students' 
evaluation, helped enormously to establish the credibility of our 
college and its programs with these groups. 

Researching Women's Abilities 

Because Alv^rno is a women's college, the research reported 
here is on women participants. Women from ages 17 to 66 are 
drawn from student, alumnae and professional groups. because of 
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the paucity of findings on women's abilities, results reported 
should be helpful to other educators and researchers who are 
attempting to understand women's abilities and develop programs 
for che large numbers of women returning to ollege. 

Using Expert Judgmer.t 
Production Measures 

Ip liberal arts colleges, expert judgment of cotoplex 
abilities is t.he primary mode of assessment. An English theme, 
history term paper. Fine Arts performance, student teaching. 
Nursing clinical, or Science lab all call for an extensive 
" production " by the student and complex judgment by an instt actor 
on the extent to which the performance meets criteria, Thi 
closer the student performance is to ^^bilities students will oe 
demonstrating across work and personal roles> the more confidence 
the instructor has that the measure is valid. Yet many paper ard 
pencil tests that call for recognitiori alone are heavily used 
because of their efficiency in administration and scoring. With 
the advent of the computer, chese measures are also more 
efficiently alidatad . 

In general, however, assessment of upper level work is Oi^ten 
far too complex for such recognition tisks, and some disciplines 
in the arts and humanities are less likely to develop and use 
measures that rely on quantitative methor^s of assessment 
developed in the behavioral sciences. It criteria for judging 
are defined to give a picture of the abilities being assessed, 
facult) can more easily discuss common abilities that c«.oss 
disciplines and set the stage for reinforcing these 
abilit ^s — like communications — across courses. And such 
measures become likely selections for a validator seeking to 
measure the outcomes of college. 

Thus, we used expert judgment in developing or choosing 
instruments fur establishing program validity. Both arts and 
^ humanities, and behavio al sciences faculty are open to 
qualitative analyses of #student responses, and expert judgment 
seems to mesh more with assessment strategies already in use. 
Faculty become more systematic and efficient expert judges very 
quickly, and are interested in specifying the basis ror judgment 
and creating criteria. We have tl ^refore adopted some measures, 
designed to be cross*disc ipl inary , as validation instruments. 
Some measures serve multiple purposes for diagnosing and/or 
credential ing student performance, and also for validating the 
curriculum, additig to instrument efficiency. 

In addition to creating and validating measures Lhey already 
use in the curriculum, faculty have been able to creete complex 
new instruments ana apply them w. h the validation team. Thus, 
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faculty in an ir: utution cen, with professional help, do much of 
the work of creating cross-disciplinary, p oduction measures of 
abilities, and also judge student performance. 

Valu ♦tMg Nontraditional Assessment Techniques 

Ii this study we validated a range of faculty-designed 
assessment techniques, and criteria and a process for judging 
student performance on the Perry scheme. We tested out a variety 
ot strategies . Current methodologies fur validating 

faculty-designed, generic ability instruments reflect a pattern 
analysis approach , rather than score analysis , correl<'*"ional 
analysis or an item analysis approach alone. These methods have 
implications for similar programs which are seeking new methods 
to establi-^h construct as well as content validity of complex 
assessment instruments . 



Testing Out New Measures of College Outcomes 

Sever i criticisms of previous' college-wide assessments like 
the Scholastic Aptitude Test include lack of relationship to * 
later careering and professional performance. Grade point 
average, aptitude and achievement scores have not predicted later 
success in some studies (McClelland, 1973). We have therefore 
used a variety of new cognitive-developmental, learning style :nd 
generic ability measures, performance interviews and inventories 
of professionals, as well as in-depth interviews of student and 
alumna perspectives to test out new methods of assessiaent . Ar 
the same time, we used some of the more traditional methods as a 
check on how results from newer asessment techniques compared. 
We found that newer methods do take more time and involvement but 
are more efficient for other reasoi The descriptions of 

outcomes these meacures yield stimui*a-e more discussion by 
faculty, have more validity for performance after college, and so 
enable colleges to establish validity tor particular professional 
areas. For example, by identifying competences that make for 
effective performance in the nursing profession across ^rious 
contexts, we can build better i: -college testing technique^ and 
at the same time, contribute to revising state board examinatiors 
in nursing to make them more per formance-^bcised . 

In general, cognitive-developmental measures, measures of 
learning styles, and some generic ability -measures proved to be 
effective measures of change during coU^g-, and our studies of 
professionals* perfon. i yielded a cadre of abilities that can 
serve as criteria ^oi < essing the performance of alumnae. This 
Ail enable us to build assessment techniques for judging 
performance interviews of alumnae, a future goal. 
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Disseminating and Implementing Results 



Research findings need to be disserainaced tc educators in a 
mode that has some practical implications for educational 
programming.' Research progress, .initial findings and problems 
had to be presented in understandable language and linked to 
practical issues important to other colleges attempting to 
improve their programs. The strategies u&ed in this project 
include: (1) Twcnty-seven presentations at state, national and 
international conferences and specially convened seminars; (2) 
Nine presentations to Milwaukee professional groups; (3) Fourteen 
research reports, eight instruments, and 20 other publications; 
(4) Six progress reports to the National Institute of Education 
and six presentations at project directors' meetings; (5) Seven 
research reports and over 20 presentations to student, alumna, 
and professional participants; (6) Thirty-three presentations to 
Alverno faculty, trustees, and ad/isory councils; (7) 
Iwenty-three presentations, with materials, to representatives of 
16b institutions or departments within those institutions who 
visited Alverno; (8) Dissemination through mailed materials to 
207 institutions and representative departments; and (9) 
Materials dissemination to representatives or at least another 
310 institutions or departments within those institutions a{: the 
27 off-campus conferences and seminars. Such dissemination 
strategies led to constant questioning of the research 
methodology and identifying those 

In addition, research findings could begin to be implemented 
through tryouts of * various instructional and assessment 
strategies. Deutsch and Guinn introduced learning styles 
assessment as a regular part of new student seminars (Deutsch & 
Ouinn, Note 1; Mentkowski & Giencke-Hol 1 , 1982); Loacker and 
Cromwell adapted criteria for judging performance on the Perry 
scheme to co*" unications learning and assessment strategies 
(Loacker & Cromwell, Note '4). Schall and Guinn used the 
Behavioral Event Interview competence assessment technique and a 
performance characteristics inventory in a project to expand 
faculty awareness of abilities professionals use on the job-. 
About 20 faculty interviewed another 130 professionals in a 
number of professional areas during the summer of 1982 (Schall & 
Guinn, Note 3; Loacker & Schall, 1983). Student perceptions were 
used to improve career development services (Fowler, Mentkowski & 
Schall, 1981). Faculty in natural sciences and technology 
designed investigative learning laboratory experiences and 
researched resulcing student performance and perceptions (Truchan 
& Fowler, 1980). Data from the longitudinal study of student 
change helped inform practice in the Office of Instructional 
S< vices (Neis^s, Note 5). 



Interview .analyses of students , alumnae and other 
professionals ha^e proved to be a rich source for cui iculum 
improvement. Results from student and aiumn^ interview analyses 
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describing the development of learning to learn are being used to 
create a new instrument for assessing students on these skills. 
Alumnae interviews have also been analyzed for clues to personal 
and professional role integration (Mentkowski & Much, in press), 
how alumnae integrated career and family responsibilities has 
implications for how faculty can assist students of all ages to 
anticipate and/or manage this dilemma while still in college. We 
are currently preparing tne 1000 critical incidents from the 
nursing and management performs'^ce interviews for u' e by faculty 
to create cases, instructional examples, and assessment 
techniques. ^ 
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SUMMARY 



This ongoing effort to enhance the quality, effectiveness and 
validity of education in the liberal arts has already contributed 
to several common objectives of college, their faculties, 
student s and al umnae , the educar ional research and eval uation 
community, outcome-centered education, as well as our own 
college . 

Ongoing research and evaluation efforts like this one and 
others can help colleges to take the initiativ to define and 
demonstrate their outcotq^es to various constituencies who ask that 
such outcomes of college meet certain standards for use. Showing 
just how the more complex thinking and problem solving abilities 
show up at work, and how adaptability in learning on the job 
functions for the new grc^duate in the business community can make 
a difference to this segment of society who has often created 
their own educational technology rather than turning to colleges 
for help. Building a bridge to the business and professional 
community in ways that show we value their input in 
education — not just for ideals bat practical abil ities — ccn 
encourage them to join forces with educational institutions. 
Demonstrating that we are willing not only to identify outcomes 
as goals, but to deal with practical realities in making college 
work for students and for the business and professional 
commijnities we serve opens up a wealth of input, particularly for 
smaller colleges. The brain drain of professors from the 
research university to corporations can be reversed at the level 
oi urcid graduate education if we tap the expert judgment of top, 
corporate personnel . 

Students and alumnae also benefit. Students benefit because 
they begin to teel ttiat education is a process. Changes can and 
do occur, and students have input into program design and 
execution. Such a model sets uj the expectation of themselves as 
change agents within the institution, and suggests a creative 
ten'^ion between the ideal and the real v^ile they are s'iii in 
college. While letting them in on the imperfect role of 
authority, it prepares them for the dynamic interplay between 
their own expectations for rhange and the conditions that are 
necessary for making changes. 

This is particularly of benefit to che cadre of new students 
who are most likely adults. Already part, of the working world, 
they come to college with a more practical stance and expect more 
concrete benefits. They are under immediate pressure to show 
family and employer that the financial and time investment is of 
benefit at work and at home while they are still in college. 
This is more critical for women since many manage multiple roles. 
And the traditional age college student, who is currently more 
focused on practical career goals, will benefit from new 



strategies that build on prior formal and informal learning 
exper iences . 



efforts like this one are expected to more directly benefit 
faculty in making it more possible for them to improve 
instruction. Most educators, pressed by the day to day 
frustrations and pressures of classroom instruction are open to 
identifying problems in teaching and looking for solutions, 
having this helpful source, in addition to others, maintains and 
stimulates their work. When results from a cross-college effort 
are available on a continuing basis, a common excitement and 
probing occurs. A collaborative sense of purpose strengthens. 



N:t to be overlooked is the bcjnefit to interdisciplinary 
discourse of insight into student develo, aewt and laarning 
processes that can cross the barriers erected by the most 
^ ndependent department . Educators need both anecdotal and 
systematic results that describe ways in which students develop 
beyond one insr i uc tor ' s class, to life after college. 



This model shows how faculty, in their roles as educators and 
instructional researchers, work together with faculty's concerns 
driving the validation effort. It enables faculty to efort. It 
enables faculty to measure things they really care to change, 
instead of measuring outcomes for which they are held accountable 
but that are not their own goals, toward wh'.ch they are not about 
change their teaching. It is also a model for devising really 
usable validation tools, because they come from the faculty's own 
goals and questions. That puts faculty in the position of being 
able to join with the administration and with the institution as 
a whole, in explaining to the rest of society what it is that 
education does. It enables the faculty to take the initiative 
and to regain the position of educating society about education, 
a task which has too long fallen into the hands of journalist.. 



There are some contributions to ability-based, 
outcome-centered education that also emerge from our work, 
first, we have felt confj.rmed in our decision some ten years ago 
to specify outcomes as a faculty and to develop curriculum 
through that common lens. We have certainly had more direct 
access to establishing the validity or our various outcomes and 
assessment process. The apparent success so far of our attempts 
to validate one laculty's ability-based approach to education 
suggests that outcome-cent ert-i education in general may indeed be 
a meaningful advance toward making college outcomes accountable. 



In doing so, we have been able to open the mo^^-e subtle 
aspects of the learning process to critique and clarification. 
New theories of learning, particularly for adults, can emerge 
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from this dialectic. Focusing on outcomes, and defining them as 
complex processes, has enabled us to link learning to learning 
goals. Identifying abilities is an activity that helps students 
to cognitively structure and organize their own vision of the 
learning process in school, c-^ves them a framework for 
establishing the relevance of lib'^ral education to their career, 
and helps them organize their careering after college. Finding 
that such abilities and procesjes transfer to their personal 
lives during college frees them to become more open to learning 
not directly relatf^ ' to a specific occupation, but to human 
growth. The effort to assess outcomes actively, as well as to 
identify them, gives the stuHent an important experiential sense 
of her own compeLence that seems to be a major catalyst in her 
development, both in school and at work after college. 



A student's ^ nse of her own proven competence becomes the 
organizing principle for her vision of her own growth and her 
strongest sense of proof. After college, it becomes a criterion 
for judging Oiether she is effectively managing her career, for 
judging IMjiether she is satisfied in her job, for making job 
changes W necessary, or for staying where she is if she is 
satisfied. This experience is so powerful in college that after 
college it becomes her major criterion for assessing and managing 
her career. 



A major reason for assessing the outcomes of college is to 
allow faculty to better accredit their stua^-nts and to allow the 
institution to be more accreditable. But we have also found that 
the experience of identifying abilities and demonstrating them 
across multiple contexts is of tremendous learning benefit to the 
student. A college that gives a student this experience is 
giving a student an advantage, whether or not outside groups 
would identify those same abilities, or judge her effectiveness 
in the same way. 



I 

ERIC 



Valuation efforts in higher education also contribute to the 
educational research and evaluation community . Program 
evaluation as a discipline is new and is currently called on to 
provide tecimical assistance in the design of large validation 
studies in a wide variety of field settings. New technologies 
must be created to meet the demand, and methods that work in some 
educational research settings do not necessarily transfer to the 
cross-disciplinary atmosphere of a liberal arts college, or even 
to more technologically-oriented universities where each 
discipline has its own well developed methodologies. It is not 
surprising, tht^n, that we would have new validation definitions 
and strategies , improved methods for conduc ting educational 
evaluations , insights into disseminating and implement mg 
results, .md specific procedures for creating Ln atmosphere of 
fairness and respect for the input of our constituencies and 
informed involvement of our participants 
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We have made advances lu the assessment of complex abilities 
including creating production measures where expert judges are 
trained to make qualitative judgments thro»»^h objective, 
analytical processes. Better ways to specify the criteria for 
judging complex abilities and tor ensuring their validity result. 

While we identify the values of validating outcomes for other 
colleges and constituencies, we trust the value to our own 
college comes through. We have made a commitment to continue our 
evaluation/ val idation process as part of the institution and a 
permanent component of the learning process. Our Office of 
Research and Evaluation is budgeted on a permanent basis, 
although staff and breadth of activity may vary somewhat 
depending on the strength of outside funding. 
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NEW DIRECTIONS 



There are several clear directions for future research and 
curricular applications. One is to probe further the meaning of 
the complex cognitive-developmental patterns, learning styles and 
personal mat uri ty/ intdrper sonal and reasoning abilities we have 
begun to study. Improved criteria for assessment , improved 
instruction, and improved educational validity of tne learning 
process will result. Researching such abilities can become a 
common task in higher education networks. we have begun to 
involve other colleges in one such effort to better define, 
instruct and assess for critical thinking (Cromwell & Allen, 



It is now clear that abilities are complex processes where 
knowledge is only one component, and that the transfer of them to 
situations during college and afterward involves .learning how to 
ada'>t abilities already learned to new situations and 
environments. Such learning on the job is much less guided and 
i; predicated by "learning to learn" or "lifelong learning," 
familiar terms but not well-researched concepts. The development 
of preferences for a range of learning styles and change in 
cognitive intellectual growth measured by the Perry scheme may be 
the first indicator that learning to learn is developing for the 
student. Huu how do these beginning preferences become 
translated into sophisticated processes, into a "theory of 
accion" toi* self-directed learning? 

We also need to make use of the patterns we have observed in 
students^ developing abilities over time to research individual 
differences in ways of learning and in the expression of those 
complex abilities. An initial stage has concentrated on 
describing broad patterns of change. It does not speak to the 
question of who changes and why. For whom is college more 
effective? Who responds better to certain aspects of the 
curriculum, who does not? Retaining students who are not 
performing is critical to the survival of many public and private 
colleges. Analyses of individual differences can specifically 
improve instructional strategies. We studied student change in a 
conservative research design to obtain a picture of tne actual 
benefits of the curriculum. We also need to compare student 
entering abilities with those of students who did not persist, to 
see what abilities predict staying in college. We have 
identified somt determinants that account for how much individual 
students benefit from the curriculum. These patterns need to be 
linked to more specific instructional strategies. 

One practical way to brild on prior efforts and to research 
I hese quest ions is to establish a system to continue to collect 
1 teaming progress information based on developmental pattern 
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data. By monitoring students* progress through the learning 
process more carefully, and building our expectations on what we 
now understand about the development of abilities, we could 
better pinpoint students who need more Support and challenge, and 
intervene more quickly than longitudinal studies permit. 

Such an »3ffort could also build on our increased confidence 
in tne validity of expert judgirent m identifying and assessing 
for abilities, and patterns in the way they develop. Qualitative 
expert judging processes can illuminate complex student thought 
processes and behavior. Assessing complex competences, 
developmental level and learning styles is possible. 
Understanding such complexities is not beyond our grasp. We 
reinforce any ^ifforts to further develop production type measures 
ard judging process s of abilities that cross disciplines in the 
liberal arts. Many of the external measures we used need further 
work to be more useful to colleges attracting working class and 
minority students. Wp need to examine in-depth the range of 
individual differences tnat occur in the normative patterns of 
change we have identified in this report. While our initial 
approach provides developmental norms for students at this 
college, and broadens the normative base for college students in 
general, it does not speak to the question of who does best in 
college and why. Retaining students who are not performing, 
particularly those who have not had strong educational 
backgrounds, is critical to the survival of many private and 
public colleges. An analysis of these individual differences, 
and the abilities we studied, will be extremely helpful to other 
colleges. Further, we used a^ conservative research design to 
measure the effects of the learning process on students precisely 
to get a better picture of the actual benefits of college. We 
studied those who graduated from college across time. We need 
also to compare the entering abilities of those who did nut 
graduate with those who did. 

Abilities demonstrated by professionals have good face 
validity with the outcomes educators usually identify. But these 
abilities, when described in a developmental sequence that ta'es 
into account the role of formal education and on-the-job 
experience, can help students in various occupational groups to 
be better 'prepared. Case studies and assessment criteria are one 
curricular application. Career advising based on professionals' 
careering histories are another. We need to continue to follow 
our alumnae and their developing per formance abilities. A 
modified format of the job competence assessment performance 
interview would be a next step in studying alumnae performance. 

Such efforts, to research the meaning of abilities, to find 
ways to incorporate professional and career development abilities 
into the curriculum, to look for individual differences in the 
way m which they develop, and to create strategies for more 
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individualized instruction are important new directions. We also 
need to study how abilities are adapted and focused through 
continued learning, to expand our description of learning to 
learn theoretically and practically. Creating a student progress 
information system that makes use of our current understanding of 
abilities, and our new assessment techniques can help to realize 
these goals. Our continuing interview research on individual 
patterns of learning and the determinants of change should also 
benefit faculty understanding of student growth. 

Finally, we need to continue our efforts to demonstrate a 
variety of validation strategies, models, methods and instrument 
designs. The fact that a liberal arts college has been 
successful in carrying out an elaborated effort, with the 
collaboration of a higher education and research network, is 
proof that colleges can develop their curriculum, do their own 
research and validation, and therefore continue to survive in 
American higher education. 



Meanwhile, our overall plan is to continue opening many of 
these issues and findings in more detail to the critique and 
comment of faculty in higher education, a process that will 
engage us and others in a renewal of interest in our chief 
concerns as educators. 



We have been excited while learning, using, and evaluating 
the concept of outcome-centered education. Alverno has been 
committed not only to designing this kind of a curriculum, but 
also to designing an intensive measurement strategy to test out 
these ideas. Ihe model presented here can be applied in ocher 
educational settings. It offers insights into new approaches in 
adult development which lay have far reaching cor»sequences in 
settings outside of higher education. In this way, we begin to 
ensure that we develop abilities that truly last a lifetime. 
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Validating Assessment Techniques in an Ojtcome-Centered 
Liberal Arts Curriculurr 
Valuing and Communications Generic Instruments 



Miriam Friedman Marcia Mentkowski 
Margaret Barley Georgine Loacker Mary Diez 



ALVERNO COLLEGE 



Two studies test methodology for validating assessment techniques in a 
performance-based liberal arts curriculum. Alverno College has a system-wide 
performance based curriculum, with an assessment process that requires students 
to demonstrate incremental gains while progressing through six sequential levels 
in each of eight competences. The oight competences are integrated with the 
concepts in each discipline. *>tudents are requir ;d to attain each competence 
Level in sequence to demonstrate cumulative achievement. These two studies 
assess the effects of instruction on patterns of student response using 
instruments. Both instruments are "generic," that is, general criteria are 
integrated with criteria specific to the way the ability appears in the 
discipline ir which the instrument is used. Studies of two generic instruments, 
assessing level 4 of the competences of Communications and Valuing are reported 
here , 

Twenty students performed on the generic Communications instrument after two 
years in college; another twenty performed upon entrance to college. They 
demonstrated abilities in four modes of communication: speaking, siting, 
listening and reading, providing data on student performance across different 
modes of the same competence. The student is also asked to self-assess her 
performance in each mode on the same criteria on which she is judged by the 
asses8or(s). Eleven students performed on the generic Valuing instrument after 
two years in college; another twenty performed upon eiitrance to college. 
Students demonstrated vflue and moral judgments and decision-making modes. 
Students also self-assess f'air parformance. 

In the Communications study, students with no' instruction Jemonstrated a wider 
range of variability in performance as compared to the instruction group, who 
showed a less dispersed pattern. Student performance varies with the mode of 
communication. The instruction ?roup performed significantly better 
particularly on the upper levels of the four communication modes. The different 
patterns of the interrelationships o2 student r^^^^ormance across the four modes 
are seen in relation to the levels. Students who had instruction can better 
self-assess their performance. 

In the V.^luing study, the instruction group performed significantly better than 
the no instruction group. Data from the instruction group provided support for 
the validity of the cumulative hierarchical nature of the competence. The no 
instruction group did not show any consistent cumulative or sequential patterns. 
Overall, the instruction group demonstrated clusters of relationships among 
scores on the criteria and the no instruction group appear.-^d to perform in a 
randomly scattered manner, indicatinp. effectiveness of instruction. The study 
methodology reflects our current pattern analysic foproach, rather than using 
score analysis, correlational analysis or an item analysis approach alone. The 
mteipretat ion of the rep Its t^nd the methodology developed have implications 
for similar programs which are seeking out new methods to establish construct as 
well as content validity of complex assessment techniques used in 
performance-based curricula in higher education. 
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Validating Assessment Techniques in an Outcome-Centered 
Libera! Arts Curriculum: 
Social inte.actlon Generic Instrument 

Miriam Friedman Marcia Mentkovv$ki 
Bernard m Deutsch M. Ntcoiette Shovar Zita Allen 

ALVERNO COLLEGE 



This report explores issues related to the validation of more nontraditional 
assessment techni<;ues, and tests some ways such studies may proceed. We 
examined the appropriateness of various methods for validating ^ generic 
competence instrument that measures Social Interaction, a construct with little 
or no history as a teachable college outcome or measure. We compared the 
performance of 69 uninstructed students on entrance to college with that of 32 
students who had two years of college instruction on each of the Social 
Interaction dimensions (Preparation, Demonstration, Self-Assessment and 
Leadership), and the specific dimension criteria. 

Results indicate similarities in performance between traditional age instructed 
students and mature uninstructed students. While this may be expected, it also 
indicates that group comparisons may not be an effective strategy for validating 
assessment techniques if the ability is one developed through prior informal as 
well as college learning. Despite our efforts to do so, we were not able to 
control for the myriad range of variables thac are likely to affect the results. 
When performance of such an ability also interacts with a set of personal and 
ego development variables, separating out the specific effects of instruction 
tha*^ show significant differences through group c'-oarison is not an effective 
strategy, especially given the small sample sizes generally available. 

However » some Social Interaction criteria did indeed separate the uninstructed 
students from the instructed students when we combined all students in a 
discriminant analysis. These criteria are more closely related to those aspects 
of Social zeraction that are learned as part of the more specific Social 
Interactio learning experiences. Thus, including studfnts with a broad range 
of age and lormal learning experience did lead to an effective strategy for 
identifying '"hose Social Interaction behaviors that validate the construct. 
Clearly» tne study of assessment techniques should not be limited to univatiate 
methods; patterns of coherent group performance provide us with a more holistic 
picture of performance, particularly of Social Interaction, not well understood 
and measured compared to some other abilities like Communications. 

The present study outlines a procedure by which the integration of info .ation 
about competence construe t , di f ferent group charac terist ics ana criteria 
evaluation contribute to an information base for instructional development, 
re~evaluat ion of competence definitions and revision of instrument criteria 
which measure these behaviors. The study helps to illuminate a key question 
approaching the validation of any faculty designed instrument measuring 
important but not well defined abilities new to higher education instruction: 
What strategies are appropriate given where this instrument and construct are in 
their current development? 
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Validating Assessment Techniques in an Outcome-Centered 
Liberal Arts Curriculum. 
Insights from the Evaluation and Revision Process 

Alverno College Assessment Committee/ 
Office of Research and ^valuation 

ALVERNO COLLEGE 



The Alverno College faculty has designed a curriculum and assessment process to 
assist students to develop and demonstrate ability in a variety of competences. 
Faculty, individually and as a group, design assessment instruments which then 
come under the scrutiny of other faculty in a continuous process of review and 
redefinition. This evaluation and revision process stimulates evaluation and 
revision of the instruments in a systematic w«y. 

Validating assessment instruments is an unusual goal for a college faculty to 
pursue. To validate means that concepts of the abilities or competences 
assessed and the means for doing so must be carefully thought out, subjected to 
rigorous reasoning, and constantly reviewed against student perfonnance 
outcomes. This report summarizes questions, suggestions, concerns and insights 
generated from feedback sessions with faculty who submitted theii instruments 
for a validation study. Sixteen instruments w^re identified by departments as 
ready to submit because faculty judged theym sufficiently developed to evaluate. 
Three validation strategies worked best of those tried. On^ is pre- ^nd 
post-instruction coranarison which determines if changes in student performance 
can be attributed to the effects of instruction. A second is criteria 
evaluation, which involved tue clarification, revision and refinement of 
criteria based on an analysis of student performance. A liiird is establishing 
Che inter-rater reliability of assessor judgments, whicn enables a test of 
reliability as well as the development of instrument criteria. Criteria 
evaluation appears to be most helpful when the instrument is being evaluated and 
revised. Pre- and post-instruction comparisons are 'sed most effectively aft 
facL Ity \ 1- dged the instrument as meeting mo3t other instrument design 
<?^l'^er*."- rter-rater reliability studies are most useful when they are 
' nduv . -urrently with criteria evaluation. The validation studies that 

we.e syn .e.,i teu for this report show that direct involvement faculty in 

ana* vziu^ L.t-.." performance data and probing validity questions generates a 
broad sec po jf ''ai:'i*ty issues. 
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Validating Assessment Techniques in an Outcome-Centered 
Liberal Arts Curriculum. 
Integrated Competence Seminar 

Alverno College Assessment Conrwnittee/ 
pffice of Research and Evaluat' < 



ALVERNO COLLEGE 



The Integrated Competence. Seminar a/*ses^smenr technique allows students to 
demonstrate integratiofi andj transfer of /^earning in three situations: Oral 
PresentaLion, In-Basket Exercise and^roup Discussion. Assessors observe and 
e aluate performance against^ spec i fiecr cr iter ia , and give feedback to students 
on their performance. A\ behavi(^al criteria checklist permits evaluation ot 
later-rater reliability, and' val ida/t ion of the technique through comparison of 
quantified assessor ludgmeiits w/th other student performance measures and a 
battery of ex^^^nal criterion pleasures administered to students in a 
longitudinal study '^f college outcomes (Mentkowski & Strait, 1983). 

Assessor judgments correlated in^ the 70 's. The In-Basket Exercise was the most 
difficult and the most valid in therms of correlation with measures of students' 
cognitive development and otl*r generic abilities. Oral Presentation showed 
mixed results, an' the Group Di/cu'sion correlated with other measures in ^^ays 
opposite to the expected dir^ttions. When age, background and college program 
are controlled, there were no significant relationships between the number of 
credits accumulate' and nu^tber of competence level units achieved. Thus, the 
In-Basket had some pe'tTcvr^=!n(/e validity, the Oral Presentation is equivocal, and 
the Group Discussion bad relationships opposite to our expectations. The 
finding on the Group Discu/sion supports our earlier findings with respect to a 
Social Interaction gener ic^instrumen t . 

Generally, the effort revrfaled that the In-Basket exercise most accurately 
measured abilities of /Analysis and Problem Solving. The Group Discussion, a 
measure of Social Interaction, worked less well. The study points to the 
importance of continvmig to develop nontraditional assessment techniques like 
In-Basket, and to revis/ the measure with particular attention to the links 
between Group Discus/ion criteria and the Social Interaction ability it 
represents. The Integrated Competence Seminar has since undergone extensive 
revisions by a group it faculty specializing in assessment design, based in part 
on the findings of tyis study. 



18:) 



ERIC 



182 



Validating Assessment Techniques ir an Outcome-Centered 
Liberal Arts Curriculum: 
Six Performance Characteristics Rating 



Alverno College Assessment Committee/ 
Office of Research and Evaluation 
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The Six PerformaTCp Characteristics assessment technique provides a means for 
faculty to judge students in a systematic way over time on developmental 
characteristics which apply to their perfonnanw*> across disciplines and across 
competence areas identified as goals of liberal learning by Alverno faculty^ 
Descriptions of six performance characteristics were prepared and first tested 
by faculty with seniors graduating in the spring of 1978. The characteristics 
were integration, independence, creativity, self-awareness, commitment, and 
habituality. The characteristics were defined by sets of descriptors for the 
''Beginning Student,** the "Developing Student," and the "Graduating Student," 
Pilot study results indicated some discriminating power (students graduating 
with honors were rated significantly higher than students graduating without 
honors). The following year fll students in the college were rated to colle':t 
additional information on inter-rater reliability, the developmental character 
of the ratings, and the extent to which the six characteristics were 
differentiated in ratings. 

Results from the first all-college administration providec* evidence of 
acceptable inter-rater reliabitity, and supported the developmental character ot 
the definitions through significant mean differences between classes. Whil3 the 
power of the measure to distinguish bttween students at different levels was 
demonstrated, it was found that all characteristics followed nearly identical 
patterns, raising further questions concerning differentiation among them. 

Six Performanc Characteristics ratings were conducted on ail classes in r 
1^30, 1981 and 1982, as part of a comprehensive program validation wh 
included other measures of student performance within the curriculum, and 
longiLudinal assessmeiits of student development and change using a battery of 
external criterion measures (Mentkowski & Strait, 1983), Ratings from the 
longitudinal study sample of two consecutive entering classes confirmed that a 
^.ingle factor accounted for 90% of the variance in ratings on each 
characteristic on three different occasions. Using the single factor, it was 
found that students were rated at significantly higher levels over time, 
corroborating the cross--sectional evidence for the developmental character of 
the procedure. The rating factor was not associated with other college 
performance measures in the longitudinal study when the influences of student 
background r.nd program differences were controlled. There was, however, 
evidence that ratings discriminated between students on academic probation and 
those wno were not, irrespective of class standing, 

Relat lo.jships between the Six Performance characteristics factor and the 
measures of human potential revealed that the faculty were making judgments 
based on a general dimension associated with several external criterion measures 
of intellectual, ego, and moral development. The strongest pattern of 
assoc lat ions was found with a measure of Perry * s scheme of intellect ua I and 
ethical development during the college years (Perry, 1970, l9?l). The Alverno 
faculty IS cont inuing to work wi th the assessment technique , at tempt ing to 
refine the definitions of several characteristics so thac a more differentiated 
picture of student development may result. 



183 




A Longitudinal Study of Student Change m Cognitiva Oev«lopm«nt, 
Ltaming Styles, and Ganoric Abilitiei in an Outcor " -Centarad 
Libarai Arts Curriculum 

Marcia Mentkowski Michael J. Strait 

ALVERNO COLLEGE 



That students change in college is taken for granted. That students change is 
the result o^ per£:>rming in a particular curriculum is more difficult to shew, 
and describing who changes and why, in relation to which complex abwiMe- is 
even mo-e illusi^^e. Thic longitudinal and cross-sectional study was designed to 
investigate three questions: Do students change in broad abilities indicative 
of human potential for cognitive-development, learning styles and other generic 
abilities? Can we attribute change to performance in a performance-based 
curriculum, rather than to age, background factors and program characteristics'^ 
What are the underlying themes or patterns of change that could be used to 
assist curriculum developers in higher education concerned with responding to 
current frameworks in adulc learning and development? 

Over >50 students participated in the longitudinal a.id cross-sectional studies 
by completing a battery of twelve instruments with developmental 
characteristics, and which employed both recognition and production tasks. The 
instruments we^e drawn princi^^ally from three ».ources: cognitive-developmental 
theory, experiential learning theory, and competence assessment desi'^ned to 
measure abilities yhich link those learned in college to profe sional 
performance afterwards' Students ranged in age from 17 to 5">; 200 fonued a core 
group for the longitudinal study using a time series desigr with assessments at 
three times during college. Change occurred in varying degrees across the 
instrument set; some of this change could be attributed to performance in the 
learning process wnen age, background and program characteristics were 
controlled. Cognitive-developmental and learning styla measures were better 
indicators of change than were the generic ability measures, su^cgesting that 
ed ".ators can measure development as an aim of higher education. As expected, 
re^gnition measures showed more cha ee than the production measure^^ Initial 
performance at entrance to college was related to a?^^ for the 
cognitive-developmental measures, and to high School grades f>r the generic 
ability measures. While more change occurred during the fir-t two ypars 
(between the entrance assessment and the one two years later), the effects of 
the learning process on studet t change were more evident during the second two 
y^rs (between the midpoint asbessment and the one two years later near i-Me end 
of college). Students appear to demonstrate two dimensions cognitive 
development, intellectual ability and soc io-eraot ic.tal maturity a. entrance to 
collage; tn?8e abilities are integrated by graduation. 

Implications for practice are that change is measurable, and that broad outcomes 
cf college can be specified and assessed. Future interpretat ionii ^'>f results 
specific to the several instruments and their interrelationships will more 
directly contribute to our understanding of the development of abilitiv^ learned 
in college. New outcome measures hav*^ been tested, and the longitudinal d?^ta 
base 01 college learning is necessary to establish relationships be^v ?en 
abilities learned in college and prof es?: lona I perfortr ice in followup studies of 
alumnae . 
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^ Justifying Learning as Relevant to Performance 

' in Personal and Professional Roles 

Nancy Much Marcia Mentkowski 
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Approaches to the study of student outcomes at Alverno include measuring 
performance in the curr iculur.., and student changes on measures indicative of 
human potential for cognitive development, learning styles and generic abilities 
(Mentkowski & Strait, 1983). This study explores sttident perspectives on 
learning as another valuable data source for validating abilities learned in 
college (Mentkowski & Doherty. 1983). How do students understand and justify 
learning outcomes? How do they understand liberal learning as relevant to 
performance in personal and professional roles^ Detailed analysis of interviews 
from 13 traditional age students at the end of their junior year were supported 
by systematic reading of 100 interviews from 37 women students interviewed 
longitudinally at the end of each college year. A qualitative methodology was 
selected that recognizes the subjective nature of the data and treats this as a 
valuable source. Systematic procedures were devised for construction of content 
patterns representing student perspectives on how they understand and justify 
learning and give meaning to day to day learning experiences. 

Two outstanding patterns consistent 'with cuiricul^r emphasis and student 
orientation appear. First, students express a career centered rationale for 
college education. Learning is justified primarily in terms of its relevance to 
practicing a particular career after college. Second is a heavy emphasis on 
learning "how-to-do" things; learning is or ought to be useful. Students regard 
the learning process as concerned with teaching them how to perform and apply 
what they know. The meaningful ness of day to day learning experiences is 
predicated upon perceived relevance of these experiences to professional 
performance. Wtiile students express dissatisfaction with learning experiences 
for which they cannot find career relevance, they succeed in developing a 
justificatory rationale for assimilating all kinds of learning including 
"wellroundedness," a variety of discipline content areas and the competences, to 
the idea of professional role performance. For these students, the competences 
are central to the structuring of learning to perform; "use" or "application" of 
learning refers to the competences. Other kinds of substantive knowledge, 
observations, ideas, concepts, theories and so on, are assimilated to the 
competences which structure learning to perform, and are linked to role 
performance. Competences offer ways of louking at things, ways of 
understanding, ways to be aware of what is important. Students experience the 
competences as meaningful and useful and anticipate their application to the 
work setting. For competences Communications and Social Interaction, for 
example, students report feelings of increased mastery, control and certainty in 
three areas that students regard as important and which are often problematic 
for young women: interpersonal relations, identity and personal choice. The 
competences support student's perception? of being more in control and more 
effective in common everyday social and work settings, including those 
t-ncountered in off-campus experiential learning settings and personal life. 
Through experiential validation of the competences, students are able to 
construct a justification for liberal learning in whicn personal growth and 
effectiveness mediate between educational experience and concepts of 
professional role performance. 
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CarBering After College. 
Perspectives on Lifelong Learning 
and Career Development 



Marcia Mentkowski Nancy Much Laura Gtencke-HoM 
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This initial study of alumnae from Alverno's outcome-centered curriculum asked 
four questions: How are alumnae learning to learn at work, and do they describe 
lifelong learning? What abilities and processes enable transfer of learning to 
professional performance and careering after college? What are alumnae 
perspectives on careering and professional development? How do the expectations 
of students and realizations of alumnae compare? We conducted followup 
interviews with 32 alumnae, and administered a questionnaire to 56; 63 seniors 
ai'jo completed the questionr«»ire , 

Interview analysis indicated that cont iauat ion of learning is a positive value, 
is part of developing competence in the job role, and is valued as intrinsically 
rewarding, which motivates career development and job choice. Learning on the 
job is based on abilities, including those learned in college. Complex 
abilities especially important for new job roles are interpersonal abilities 
learned in college. They were strongly emphasized among both younger and older 
women and across all professional groups, as an important foundation for both 
performance and continued learning. Reasoning abilities such as analysis, 
problem solving, decision making, planning and organization also transfer to 
work These abilities are integrated and overlapping in practical role 
performance. Learning on the job, apart from additional technical learning, 
involves further development of major abilities and their integration and 
adaptation in relation to work contexts. Learning how to adapt abilities 
involves a process of applying judgment and abilities in action, getting 
feedback and adjusting accordingly. Learning to learn means discovering how to 
derive from an environment and experience what one needs to know to adapt one^s 
abilities. Most women viewed work through some concept of careering, looking 
beyond the present job to a professional future. Professional ideals were 
important in relating to work. Older women had a specific direction to 
long-range career goals; younger women were more immediately focused on 
development of competence in their present jobs. Career satisfaction was 
strongly related to experiencing competence on the job. Satisfaction with 
ability to do a job well is fundamental for careering. A feeling of persistent 
inadequate performance of the job role led to change of jobs or career. Such 
changes re-establish a feeling of professional competence. Work satisfaction 
involved job enjoyment, a sense of relaxation and being comfortable with work; 
and progress. All women had strategies for career progress, but older women had 
more complex and long range career strategies than younger women, who focused 
more on excellence now. 

The cross-sectional questionnaire study found that seniors expect to work after 
college; 96 percent of alumnae sought work, 92 percent succeeded, 89 percent 
found work directly related to their major. These women had more professional 
positions than their mothers. Seniors had higher career expectations than 
alumnae were able to realize after two years, but alumnae rated satisfaction 
with a first position and potential for advancement as above average. Alumnae 
show more positive attitudes toward college learning after two years than 
seniors; both rated it above average. Forty-one percent of alumnae reported 
additional education; 56 percent said they planned more. Alumnae attribute more 
importance to educational goals than graduating seniors; both said they achieved 
their important career and personal goals. Older alumnae view analysis and 
self-directed learning as more important than do other groups. Potential for 
advancement is powerful in determining career satisfaction. 
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Developing a Professional Competence Model 
for Nursing Education 



Marcia Mentkovvski Vivien DeBack 
James M. Bishop Zita Alien Bart^ara Blanton 
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The major purpose of this study was to create a generic competence model for 
effective nursing performance. The major outcome is a codebook describing nine 
generic abilities. The competences were derived after an intensive qualitative 
analysis of performance interviews from 80 outstanding and good nurses in which 
nurses discussed what they actually did in situations that led to effective and 
ineffective outcomes. A peer nomination questionnaire yielded outstanding and 
good groupings of nurses; a background questionnaire provided information on 
education and experience. Hurses were employed in a long-term care setting, an 
acute care setting and a community health agency. 

Nurses perform a great deal of Helping, a competence which fits with the more 
traditional role of the nurse. But they also perform Independence, Influencing 
and Coaching to a large degree, and they perform Conceptualizing. These 
competences describe today*s nurse as an active, influential professional t/ho 
demonstrates independence and analytical thinking in her role. More of these 
active competences were demonstrated in the community health agency than in the 
acute care agency; the acute care agency and the long-term care agency seem to 
have a more structured environment with regard to roles and tdsks. Nurses in a 
more structured situation may not demonstrate some of these abilities to a 
greater degree because of the demands of the setting. 

The more experienced or more educated nurse is likely to demonstrate more 
Conceptualizing, less negative Conceptualizing, more Ego Strength, and more 
Independence, Influencing and Coaching. These competences taken together seem 
to have an underlying component — an active, thinking, influential style wh«re 
the nurse also strives to assist the client to take on more responsibility for 
his or her own care. Seme of these abilities aopear more in the' community 
agency, an agency we believe is likely to be more supportive of these 
competences, where more educated nurses are employed, and where nurses are 
likely to have more role autonomy. 

This study contributes to efforts by nursing associations and educational 
programs to assess effective nurse competences. In this study, nurse educators 
and nurse practitioners were able to cooperate in a common effort to develop a 
competence model that can improve nursing education. The 350 situations 
described by the nurses in the performance interviews can also serve to improve 
case study and other instructional and assessment materials. Nursing curriculum 
needs to build on the performance abilities of effective nurses. 
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Developing a Professional Competence Model 
for Management Education 



Marcia Mentkowskt Kathleen O'Brien 
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This study identifies abilities or competences that ensure effective managerial 
performance and sequences them to create a model of effective managerial 
performance . Performance, perceptions and career ing and professional 
development of 103 women managers and executives from 53 Milwaukee private 
corporations are described and related using a recently developed performance 
measurement system. Three outcomes result: a competence model of effective 
managerial performance for improving management programs ; a pool of over 500 
behavioral examples set within particular contexts that can be used in 
instruct ion and assessment ; and better advice for women students seeking 
examples of careering and professional deve lopment and how it relates to 
effective performance in the managerial role. 

No one competence dominates the performance of these managers. They demonstrate 
abilities across the broad spectrum of interpersonal, intellectual, 
entrepreneurial and soc io-emot ional abilities. Women managers demonstrated 
.:^nte 1 lec tua 1 and entrepreneurial abilities to the same degree as they 
demonstrated interpersonal abilities. Educators creating sequential management 
cjrricula and managers planning their own professional development can benefit 
by knowing whether some competences are prerequisites for others. Several 
factor, cluster and path analyses were performed. Competences are in »:he main 
independent of each other but some are bedt learned in sequence. A manager^s 
ability to initiate rests on intellectual skills; ability to get the job done 
rests on people skills. Underlying these is self-assessment, the ability to 
learn from -one's experience. 

Abilities effective managers judge as critical to outstanding performance are 
generally the ones they perform in day to day situations. Two abilities 
important to outstanding performarce according to managers 'and that were not 
performed often in this study are using networking and negotiating wih-win 
situations. Demonstrating self-control and positive regard for others, 
abilities demonstrated often, are apparently more critical to effective 
managerial performance than managers judge them to be. 

Implications for management education are that programs teach and assess for a 
range of complex abilities. Traditional management education has focused on 
developing particular technical skills yet specialized knowledge did not play a 
critical or decisive role in the situations described by these effective 
managers. Education that prepares for the future will include learning to 
integrate abilities, to test them out in a range of work situations and to 
critically appraise one's owii performance. Both work environment and job 
function affect the extent to which these abilities are demonstrated; this 
suggests that adaptability of one's abilities is critical for effective 
performance. There are, however, a common set of broad competences educators 
can expect will generalize across situations and contexts. Abilities on which 
the Alverno program is built mesh with those demonstrated by effective managers. 
The study provides a cadre o.f interview material for building realistic and 
relevant instructional experiences, a model for sequencing competences, and 
insights into careering for structuring career development activities. 
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Using the Perry Scheme of inteiiectual and Ethical Development 
as a College Outcomes Measure 
A Process and Criteria for Judging Student Performance'^ 

Vols. I & 11 

Marcta Mentkowski Mary Moeser Michael J. Strait 
ALVERNO COLLEGE 



This study describes use of the P«»rry scheme of intel lec "rual and ethi A 
development as a broad measure of growth m college in an outcome-centered 
curriculum. Issues that arise m applying cognitive-developmental measures to 
validate a learning process in relation to students' projected potential for 
development during college are discussed, as are those questions educators raise 
in applying and measuring the scheme, examines how colleges can demonstrate 
change as a result of curricula, improves adequacy of judgments made in relation 
to the scheme, and identifies issues educators need to consider in usii.g the 
scheme appropriately. 

The report's major focus is description of the criteria and process that yields 
judgments of student performance relative to the Perry scheme. The criteria 
(descriptive statements) . and judgment proceas, together with a set of 46 
examples showing how the criteria are applied independently by assessors and 
through consensus, should assist other persons to analyze student performance 
relative to the scheme. Reliability of the process fdr assessment and validity 
of the criteria and the ir.strumen*: stimuli and mode are examined in relation to 
assessor decision-making and judgment, and student performance on essays. The 
study underscores the importance of continuing to research expert judgment as a 
technique for assessing student performance in college. 

Parts of Volume I and assessor training materials in Volume II can serve as 
training and rating manual. Volume I contains the process for judging student 
performance on the Perry scheme, the Alverno criteria used in the judging 
process, and documents steps taken to establish 1) reliability and validity of 
the judging process and criteria, and 2) validity of the Alverno criteria in 
relation to their use by assessors. Data reported contribute to establishing 
face, psychometric, criterion group, longitudinal, convergent/divergent, and 
educational validity of the Measure of Intellectual Development (MID), This 
measure, based on Knefelkamp and Widick's work, has a history of research and 
use to which this report contributes. 

Persons new to the Perry scheme, with little if any background in developmental 
psychology and theories of assessment did learn to rate essays at satisfactory 
levels of inter-judge agreement prior to consensus. Agreement prior to 
consensus increased during the :raining sessions from 57Z to 65t to 78Z, which 
we believe resulted from concurrent improvements in four phases of criteria 
development. Inter-judge agreement on final rating of almost 3000 essays was 
76Z prior to consensus, a percen*- reached by a new assessor trained in the 
judging process. 'Inter-judge agreement with an expert external assessor was 67% 
prior to cv^nsensus. Analyses of almost 20,000 assessor )udj?raents showed that 
the themes of the developmental scheme were found useful in judging essays. 
So.ue criteria were lyaed more than other*. Criteria of a ger»eral and specific 
nature were equally usefuK Criteria from position 2 "What to Learn," position 
3 "How to Learn" and position 4 "How to Think" were used most. Criteria that 
describe aspects of a stable position are distinguished from those that describe 
the dynamics of transition between positions. Thus, the criteria describe the 
evolution of student change. Generic criteria are distinguished from those 
specific to essay type, and suggest applicability across other essay types or 
performance modes like the interview. Efcsay specific criteria show how the 
content of an essay interacts with underlying structures in development. 
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Volume I also describes results from a five-year longitudinal study of stjdent 
development on the Perry scheme (Mentkowski, Moeser & Strait, 1983), including 
relationships to ocner cognitive-developmental mei»8ures (Kohlberg, Rest, 
Loevinger, Piaget). Participants were 750 women aged 17-55. Applying the 
method and criteria, we found that the measure shows definite change in bolh 
cross-sectional and longitudinal studies. The criteria and process did work to 
desc r ibe di f ferences between students and student change over time . The 
patterns of change, however, are the intriguing results. Each of the three 
essays, Best Class, Decision and Career shows change. But the change is not 
always straightforward. When development occurs depends on the area. Rate of 
development is related to age for decision-making and career understanding at 
entrance to college, but not for students' understanding of classroom learning 
processes and roles. But after two years, older studentt. have made more 
immediate progress m understanding concepts such as learning through multiple 
ways , 1 earning f rom peers , and becoming independent at one * s own learning . 
Formal learning experiences are necessary for enhanced understanding of these 
concepts. Student change on any of the three areas of development is not 
related to high school grade average when students enter college, nor does it 
account for change during college. Students change on the Perry scheme, and 
development is differential depending on the area of development. 

These results illuminate the way students change in college, and examine the 
issue of the contribution to student development by the college experience for 
both traditional and non-traditional aged students. This study points to the 
need for careful translations between any theoretical model of development and 
Its adaptation for program evaluation, instruction and assessment, and further 
theory building. 
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DISSEMINATION 



We disseminated progress and outcomes ot the research 
reported in thii> overview and summary with several strategies. 
Altogether, the dissemination eftort yielded o4 presentations to 
persons from othei institutions in education, government ana 
business, wnich together witn materials mailed, reached bbb 
institutions and representative departments trom ail b<j states 
and from IJ countries. An additional 5J presentations were made 
to Alverno attiliated persons. We also created or contributed to 
publications. 

Specifically, we (1) made 27 presentations at state, 
national, and international conferences, and .spec lally convened 
seminars; (2) made nine presentations to Milwaukee protessional 
groups; (3) created 14 research reports, eight instruments, and 
contributed to or created 20 other publications that brought 
requests tor more information; (4J prepared six progress reports 
tor the National Institute oi Education and made six 
presentations at project directors' meetings that identitied 
problems and issues encountered in carrying out the research; (3) 
created csven research reports tor students, alumnae, and 
protessional participants, and made over 20 presentations to 
them; (b) discussed progress, procedures and results in Ji 
pre sent at ions to Alverno tacul ty , administrators, trustees , 
college committees, departments and divisions, and outside 
advisory councils; (7) made 23 presentations at Alverno College' 
workshops and Seminar Days, where we distributed materials and 
discussed several issues described in this overview with 
representatives trom 168 institutions or departments within those 
institutions; (8) mailed requested materials to persons at 2u7 
institutions and representative departments; and (9) disseminated 
materia] s to representatives ot at least 510 ot the many 
institutions or departments within those institutions (partial 
listing as registrant lists were otten not available) attending 
the 27 ott-campus conferences and seminars at which research 
findings were presented. Strategies are summarized below. 

• Twenty-seven pr«9entations at state, national, Brtd International conferences and 
specially convened seminars 

• Nine presentations to Milwaukee professional groups 

• Fourteen research reports, eight instruments, and 20 othei publications 

• Six progress* reports to the National Institute of Education and six prcst'ntat ions' 
at project dlrectois* meetings 

• Seven research reports and over 20 presentations to s*^udent, alumna, and 
professional participants 

• Thirty-three presentations to Alverno faculty, trustees, and advisory councils 

• twenty-three presentations, with nateTlals, to representatives of 168 institutions 
or departments within those institutions who attended Alverno 

• Di *5semina t ion through mailed materials to another 207 institutions and representative 
departments 

• Mdteriala dissemination to representatives of at least snochcr 510 Instleutlons or 
departments within those institutions at the 27 off-can>pus conferences and seminars < 
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Presentations at State, National and 



International Conferences and Seminars 



Throughout the grant period and afterward, we presented tne 
research rationale, progress and preliminary and final results at 
slate, national, and international meetings and specially 
Convened seminars. Participants at these meetings included 
researchers, persons in business and industry, college teachers, 
staff and administrators, officials of foundations and 
associations for colleges, officials of ttie Department oi 
Education, NIL, Fund for the. improvement of Post-Secondary 
Education, National Science foundation. National F.ndowment for 
the humanities, and other private and public agencies. 

Mentkowski, M. College as an enabling institution: Moral and 

intellectual development in the college years . Presentation at 
the annual meeting of the Association for ^toral Education, The 
Ohio State University, Columbus, OH, November 19b4. 

Mentkowski, M. D eveloping the valuing ability through the college 

curricul um. Presentation at the annual Student Affairs Institute: 

Promoting Ethical Development in College Students, Iowa Stale 
University, Ames, lA, July 1984. 

boherty. A., Loacker, G. , & Mentkowski, M. Assessing and validating 
1 e a r n i ng out c om e s . Presentat i.on to the University of New Mexico 
School of Medicine and the University of Montana School of Law, 
Albuquerque, NM, April 1984. 

Mentkowski, M. , Doherty, A., & Read, J. Abilities that last a 

lifet ime , Invitational presentation to selected attendees at the 
annual meeting of the American Association for ,higher Education, 
Chicago, IL, Marcn I9b4. 

Mentkowski, M. , Dotierty, A., & Read, J. Abilities that last a 

lifetime: Outcomes of the Alverno experience . Presentation at 
the Brookings Institute, sponsored by tee National Institute of 
Education and the American Association for higher Education, 
Washington, D.C., January 1984. 

Mentkowski, M. , boherty. A., Loacker, G. , 6i Read, J. Developing 
abilities that last a lifetime . Presentation at the Carnegie 
Corporation, sponsored by the American Association for Higher 
Education and the Carnegie Corporation of New York, New York, 
December 1983. 

Ment kows ki , M . Cognitive development in the. college years: ihe 
Perry scheme and reflective judgment . Presentation with Blythe 
Clinchy, Mary Brabeck and Karen Kitchener at the annual meeting 
of the Association for Moral Education, Boston, MA, November 1983. 

Mentkowski, M. Guiding the student toward becoming an independent 
learner . Presentation at the annual meeting of the National 
Academic Advising Association, St.^I^if^s, MO, October 1983. 
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Mentkowski, M. Assessing exponential iearriing . Presentation at 

the Council for" the Advancement of Experiential Learning Institute: 
Learning and Personal Development: The Synthesis of Knowledge and 
Expe lence, Georgetown University, Washington, D.C., August 1983. 

hentkowski, h. Can the concept of human development supply a 
unifying purpose for higher education ? Presentation at the 
Conference on Education and Training tor human Development, 
Memphis State University, Memphis, TN, Juue 1983. 

Mentkowski, M. Is there life after college? Establishing the 
validity of college-learned abilities fof later careering and 
profes sio nal performance . Presentation at the Eleventh 
International Congress of the Assessment Center Method, 
Williamsburg, VA, June 1983. 

Mentkowski, M. , & Strait, M. A longitudinal study of change in 

cognitive development and generic abilities in an outcome-centered 
liberal arts curriculum . Paper presented at the meecing of the 
American Educational Research Association, Montreal, Canada, 
April 1983. 

Mentkowski , M. , Moeser, M., & Strait, M. Using the Perry scheme of 
intellectual and ethical development as a college outcomes measure : 
A process and criteria for judging atudent performance . Paper 
presented at the meeting of the American Educational Research 
Association, Montreal, Canada, April 1983. 

Mantkowski, M. Student development on the P^ rry scheme . Presentation 
at the annual meeting of the Association for Moral Education, 
Minneapolis, MN, Noven^ber 1982. 

Mentkowski, M. Current uses of experiential learning theory at 
Alverno College ^ Presentation at the Brain and Learning Styles 
Conference, Chicago, IL, October, 1982. 

Strait, M. A study of college outcomes . ^Present ation to the Illinois 
and Wisconsin Association of Registrars and Admissions Officers, 
RockforJ, IL, October 1982. 

Mentkowski, M. Issues in prufatam evaluation . Workshop at the Second 
Conference on Ceneral Education at the Inter American University oi 
Puerto Rico, San Juan, Puerto Rico, September 1982. 

Mentkowski. M. , & HcEachern, W. De veloping a professional competence 
model for management education . Presentation to the Tenth 
International Congress of the Assessment Center Method, Pittsburgh, 
PA, June 1982. 

O'Brien, K. DeveiQping compecences for a business and management 
program . Workshop for the annual meeting of the Organizational 
behavior Teaching Society, Cleveland, Oh, Summer, 1982. 
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Mentkowski, M. Using the Perry scheme of intellectual and ethical 
development as a college outcomes measure . Paper presented at the 
"Whence and Whither'' Perry Conference, Augsburg College, Minneapolis, 
MN, June 1981. 

Friedman, M. Validatiug change in student outcomes . Presentation to 
the Wisconsin Association for Collegiate Registrars and Admissions 
Officers, Sheboygan, Wl, (jctober ' 1980. 

Loacker, G. , & Mentkowski, M. Establishing educational competence usin g 
assessment center methodolo^ at Alverno . Presentatio« to the Eighch 
International Congress of the Assessment Center Method, loronto, 
Canada , June 1980. 

Mentkowski, M. , DeBack, V. j Bishop, J., Allen, Z., & blanton, B. 

Developing a professional competence model for nursing education . 
Paper presented at the meeting of the American Educational Kesf rch 
Association, Boston, April 1980. 

Friedman, M. , & Mentkowski, M. Validation of assessment techniques in 
an outcome-centered liberal arts curriculum: Empirical illustration s 
Paper presented at the meeting of the American Educational Research 
Association^ Boston, April 1980. 

Mentkowski, M. Ihe design and assessment of an undergradu ate curriculum 
whose goal is the development of valuing . Paper ^resented at the 
meeting of the Association for Moral Education, Philadelphia, PA, 
November 1979. 

Meatkowski, M. Research implications and results from a study ot learning 
styles and professional competences. In A. Wutzdorff (Chair), Learning 
to learn at work: Case study, implementa tion model, researc h 
impl ications . Symposium preset ted at tw meeting of the Council for 
the *V*vancement of Experiential Learning, St. Paul, MN, April 1979. 

Mentkowski, M. A research and evaluation model for validating the 

abilit ies learned in college tor later success . Paper presented at 
the Sixth international Congress on Assessment Center Method, WtiiLe 
Sulfer Springs, WV, June 1978. 

Presentation to 
Milwaukee Professional Groups 

Another strategy for dissemination ca' _s for presentations 
ana distribution of materials to community professional groups. 
Most of these groups consist o ^ representatives of various 
professions. H^e following list is indicative of the range of 
groups to whom we disseminated information £nd materials: 

Presentations of the nursing study by Vivien DeBack, 
Nursing Chairperson, to Milwaukee. professional community 
groups : 
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• Greater iMilwaukee Ar«a Nursing Service, i-96U 



• Nursing Education Adr inistrators Oroup, ^^60 

• Nursing ^ministrators of th'S Visiting 
Nurses Association, 198U 

• Sigma Ilieta Tau, a national nurses^ 
honor society, ISoO 

Presentations of the studies of alumnae and professional 
managers and nurses presented by Marcia Mentkowski to 
the following groups; 

• Legal Auxiliary of Wisconsin, June 1980 

• Association for Women Lawyers, 
Decen'ber 1981 

• Inter-Groap Council, a group of 
professional women , Junt 1982 

Presentation of results frcxn the management study by 
Ilarcia Mentkowski and Kathleen O'Brien were made to: 

• Professional Dimensions, a group of 
professional women, including management 
study participants, February 1983 

Presentations of alumnae study of the integration of 
career and family by Marcia Mentkowski: 

• "Work and family: How can 1 do both?" 
Alverno Cf>llege lelesis series: Building 
on our ex ^^erience : Wom^n talking with 
women , 1982, 1933, 1984 

Publications 

The 14 research reports and eight instruments are listed on 
pages 45-49. The following are additional publications where the 
research has been disseminated; either the findings were the 
central subject, or are referred to in the publication. 

The article describing the major findings, "Abilities Ibat 
Last A Lifetimes Outcomes of The Alverno Experience" appeared- in 
the February 1984 AAHE Bulletin (official publication of the 
American Association for higher Education distributed to 6,500 
educators). Notice of the research findings appeared in the New 
York Times, USA Today and The Milwaukee Journal in late 1983 and 
T984. 
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Miller, F. American Association of higher Education features Alverno. 
Aiverno loday , May 1964, pp. 1 and A. 

Lewis, J. A college capitalizing on student abilities. The Nev York 
Times, Section 12, April 15, 19b'4. ^ 

Letson, Research project shows Alverno* s "Ability Curriculum** 

deserves the attention it's getting across tne country. 
School Sisters o ^ St . Fra ncis; United St ;.t^s^ 3(2), April 1964, 
p. 3. ' ^ 

Do abilities learned in college irake a ditference? Alverno, 
Winter 1964, pp. 2-5. ~ 

Loackei-, G. , Cromwell, L. , Fey, J., & Rutherford, D. Analysis and 
communication at Alverno: An approach to critical thinking . 
Milwaukee, Wl: Alverno Productions, 1964. 

Green, S. A college whose time has come! Al pha , 6(6), harch 1964, 
p. 1. 

Mentkowski, M., & Doherty, A. Abilities that last a lifetime: .utcome 
ot the Alverno experience. AAHE Bulletin , 35(6) , February 1964, 
pp. 5-6 and 11-14. 

Wisconsin college teaches thinking, problem-solving . USA To day, 2(94), 
Friday, January 27, 1964. ^ — ^ 

bednarek, D. I. Alverno scores well in study. Ihe Milwa ukee 
Journal . luesday, December 13, i'»63. 

Office of Research and Evaluation repoits findings. Alverno 
Educators* Newsletter . Alverno College, June 1963.""""^*"^ ^ 

Alverno research identifies nurses* abilities. Forward, 12, 1961, ^ 
pp. 49-50. — 



Hechinger, F. Women^s colleges going strong. Ihe M i lwaukee Jour nal, 
Sunday, hay 3, 1981, p. 10. ~ ~^ 

Larley, M., Mentkowski, M., & Schafer, J. Valuing at Alverno: Ihe 
valuing process in liberal education . Milwaukee, WI: Alverno 
Productions, 1960. 

Ihe Alverno valuing program: Jennifer tells impact of program. 
Alverno Today , Winter, 1980, pp. 4-6. 

Mentkowski, M. Creating a "mindset** for evaluating a liberal arts 
curriculum where valuing i^ a major outcome. In L. Kuhmerker, 
M. Mentkowski & V. L. Ericksou (Eds.), Evaluat ing moral development 
and evaluating educational programs that have a value dimension . 
Schnectady, UYi Character Research Press, 1980. 
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Valuing Competence Division, Alverno College. Understanding the 

student's perceptions of her developing valuing ability: Interviews 
with "Jennifer" through her college years . Milwaukee, ^^1: Alverno 
Productions, 1980. (Videotape) 

best nurses: Strong, caring. The Milwaukee Journal , Sunday, 
June 15, 19b0. 

Alverno College Faculty. Assessment at Alverno College . 
Milwaukee, Wl: Alverno Productions, 1979. 

Alverno awarded NIE grant. Alverno Today , Winter, 1979. 

Alverno awarded research grant. The Milwaukee Journal , ^k?vember, 1^77. 

Progress Reports to the 
Nationa l Institute of Education 

hentkowski, M., & Doherty, A. Careering after college: Establishing 
the validity of abilities learned in college for later success . 
First progress report submitted to the National Institute of 
Education, February 21, 1978. 

Mentkowski, M. , & Doherty, A. Careering .ter college: Esta blishing 
the validity of abilities learned in college for later success . 
Second progress report submitted to the National Institute of 
Education, July 15, 1978. 

Mentkowski, M., & Doherty, A. Careering after college: Establishing 
the validity of abilities learned in college for later success . 
Year-end progress report submitted to the National Institute of 
Education, January 30, 1979. 

Mentkowski, M. , & Doherty, A. Careering after college: Establishing 
the validity of abilities learned in colleg.: for later success . 
Mid~year progress report submitted to th»- National Institute ot 
Education, July 30, 1979. 

A 

Mentkowski, M. , & Doherty, A. Careering after' college: Establishing 
the validity of abilities learned in college for later success . 
Second year-end progress report submitted to the National Institute 
of Education, January 30, 1980. 

Mentkowski, M. , & Doherty, A. Careering after college: Establishing 
the validity ot abilities learned in college for later success . 
Final progress report submitted to the National Institute of 
Education, September 30, 1980. 
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Reports to Student, Alumna 
And Hrofr^aional Participants 



Communications to students took the form of oral 
presentations and letters throughout their participation in the 
study. In addition, we provided students with tour wtitten 
reports (In Mentkowski & Strait, 1983, Appendix II); they 
received a fifth as alumnae (see below). 

Mentkowski, M. Changes in student profiles on the Learning Style 
Inventory . First report to participants in a longitudinal study 
of college outcomes. Milwaukee, WI: Alverno Productions, 1981. 

Mentkowski, M. Understanding the development of thinking in college . 
Second report to participants in a longitudinal study of college 
outcomes. Milwaukee, WI: Alverno Productions, 1981. 

Mentkowski, M, Some questions and anSvers about evaluation studies . 
Third report to participants in a longitudinal study of college 
outcomes. Milwaukee, WI: Alverno Productions, 1979. 

Mentkowski, M. , & Bowler, D. Learning to learn at work: Students , 
alumnae and other professionals . Fourth report to participants in 
a longitudinal study of college outcoines. Milwaukee, WI: Alverno 
Productions^ 1981. 

Alumnae and participants in the professional studies in 
nursing and management and alumnae studies were mailed reports. 
Names of institutions are not listed to preserve confidentiality. 

Mentkowski, M, , DeBack, V., Bishop, J., Allen, 2., & Blanton, B. 

Developing a P^^of^ssi^Q^al comj>etence model for nursing education . 
Milwaukee, WI: Alverno Productions, 1980. 

Mentkowski, M. , O'Brien, K. , McEachern, W., & Fowler, D. Developing 
a professional competence model for management education . Final 
report summary for participants, Milwaukee, WI: Alverno 
Productions, 1983. 

Mentkowski, M., Much, N. , & Giencke-Hol 1 , L. Careering after college . 
Fifth progress report to participants in a longitudinal study of 
college outcomes. Milwaukee, WI: Alverno Productions, 1984. 

Presentations to Alverno Faculty , 
Administrators, Trustees, and Advisory Councils 

Communications by the principal investigators to faculty 
about the rationale for the study, progress reports and results 
were made through oral presentations, memos, and copies ot 
materials sent to students (Mentkowski & Strait, 1983, Appendix 
II). Presentations by the Director of Research ana Evaluation 
highlighting particular aspects of the work were made to the 



ERIC 



200 

206 



cotpi^rate taculty at the semester end institutes, and to new 
faculty each year who were given an overview of the study during 
New Faculty Orientation. Eight such presentations were made to 
the total faculty on the purpose, rationale and progress of the 
research. Thirteen more specific reports were made to various 
departments or committees in the college. Hie Director assisted 
Alverno' s Career Development staff to instruct faculty in the use 
of the behavioral event interviewing technique for a summerj 1982 
project to gather information about job abilities as they relate 
to careers. 



Hiree presentations were made to the Board ot Trustees; the 
first dealt with the results of studies of student perceptions 
(Fall 1977); the second reported on the purpose, rationale and ' 
progress of the study of managers (Spring 1980), the third was 
made on the major findings (Spring 1984). 



There were several reports to advisory counc ils . They 
include reports to the Evaluation Advisory Council: October 
1978, February 1979, November 1979, June 1980, April 1981, April 
1982, and March 1983; and reports to the Management Advisory 
Council: June 1979, and September 1979. Individual sessions 
with Ad^'ib :y Council members were held in 1981, 1982, 1983 and 
1984. 



An effective form of dissemination to persons outside Alverno 
occurred at the twenty-three Seminar Days and workshops attended 
by representatives from 168 colleges, universities (or various 
aepartments within those institutiona) , and other organizations. 
A review of the list o£ institutions includes colleges and 
universit ies , corporations . and other private and pub 1 ic 
institutions. Many of these institutions sent a number of 
representatives to the specialized workshops for college teachers 
in assessment and valuing, and to Seminar Days. We b<*lieve this 
is an indication that these" institutions are interested in 
improving practice in higher education and are willing to make a 
long-range commitment. 



Seminar Day on High Performance Learning is a on-^-day session 
tor persons interested in an overview o f outcome-centered 
learning and assessment. Attendees receive a one~hour 
presentation from the Director of Research and Evaluation on the 
validation design and the research findings. The Assessment 
Workshop for College Teachers and the Valuing Workshop tor 



Dissemination to Representatives of Institutions 



Who Visited Alverno 



Seminar Days and Workshops at Alverno 
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College Teachers are one-week workshops. In the first, the 
Director presents an overview and summary; in the second , 
insights from research efforts are introduced as they apply to 
various areas under discussion. Materials are distributed to 
attendees and participants also may select from available 
reports. Our policy was to initiate and maintain a network of 
relationships in the research and higher education community to 
help us adapt methods and develop instruments and procedures to 
meet the research objectives. Part of this network was created 
by the contacts made through early dissemination of our efforts. 



There were 36 institutions in Wiscoiisin tl.al participated, 
which includes 15 of the institutions in higher education in this 
state. One hundred twenty-four institutions and organizations 
participated at the national level, and eight at r.he 
international level. It was these presentations with questions 
and discussion that most clearly focused our work, and was the 
most effective strategy for dissemination. With this final 
report, we expect to reach a wider variety of audiences who are 
concerned and committed to the validation and evaluation of 
higher education programs. The 168 institutions and 
presentative departments to whom we disseminated research 
tcomes and materials at the state, national, and international 
level follows (the nuraber following a listing indicates the 
number of representatives attending) . 
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Institutions Receiving Office of Research and Evaluation 
Presentations and Materials at Alverno 
Seminar Days and Workshops 



Wisconsin 

AUis Chalmers 
Milwaukee, Wisconsin 

Appleton Electric Company 
South Milwaukee » Wisconsin 

Archdiocese of Milwaukee 
Milwaukee » Wisconsin 

Audubon Middle School 
Milwaukee, Wisconsin 

Carthage College 
Kenosha, Wiscoasin 

Edgewood College 
Madison, Wisconsin 

iJeerfield School 
Deerf^eld, Wisconsin 

Oesu Church 
Milwaukee » Wisconsin 

Immaculate Heart ot Mary Parish (2) 
West Allis, Wisconsin 

Inruadi, Inc. 
Milwaukee, Wisconsin 

S. t. Johnson & Son» Inc. 
Racine, Wisconsin 

Marian College of Fond du Lac 
Fond du Lac » Wisconsin 

Marquette University (3) 
Milwaukee, Wisconsin 

Medical College ol Wisconsin 
Milwaukee. Wisconsin 

Milwaukee Area Technical College 
Milwaukee, Wisconsin 

Milwaukee School of Engineering (2) 
Milwaukee, Wisiionsin 

Milwaukee Urban League 
Milwaukee, Wisconsin 

New Berlin High School 
New Berlin, Wisconsin 



mc 



to 1984 



Northwestern College 
Watertown, Wisconsin 

Pius XI high School 
Milwaukee » Wisconsin 

St. Alphoasus School 
Greeadale, Wisconsin 

St . Frederick Parish 
Cudahy, Wisconsin 

St . Gregory Parish 
Milwaukee* Wisconsin 

St . Joseph Convent 
Milwaukee, Wisconsin 

St. Joseph High School 
Kenosha, Wisconsin 

United Community Center 
Miliiaukee, Wisconsin 

University oi Wisconsin - Green bay 
Green Bay, Wisconsin 

University of Wisconsin - Madison 
Madison, Wisconsin 

University of Wisconsin - Milwaukee (3) 
Milwaukee, Wisconsin 

University of Wisconsin - Oshkosh (2) 
Oshkosh, Wisconsin 

University of Wisconsin - Parkside 
Kenosha, Wisconsin 

University of Wisconsin * Platteville (2) 
Piatteville, Wisconsin 

University of Wisconsin - Whitewater 
Miitewater, Wisconsin 

Washington High School 
Milwaukee , Wisconsin 

Wiscoasin Lutheran high School 
Milwaukee, Wisconsin 

Wisconsin State Council on Economic 

Education 
Milwaukee, Wisconsin 
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Nat tonal 

A Consult at ion Consort iim for 

Orgtniztt lontl Developnent 
Vernon Hillt, IlHnoit 

Alaska Department of Educction 
Jonetu, Alaska 

Alaska Pacific University 
Anchorage, Alaska 

Alvernia High School (2) 
Chicago, lUinoia 

Anna Harie College 
Paxton, haaaacbuaetta 

Association of Catholic Colleges 
a!4d Ubiveraitiea 

Washin((ton, D.C. 

Baldwin-WallGce College (3) 
Berea, Ohio 

Barat College (2) 
Lake Foreat , Illinois 

Barry College 

Him Shores, Florida 

Bay de Noc Commtaiity College 
Escanaba, hichigan 

Bellevue College (2) 
Bellevue, Nebraska 

Belloiont College 
NashvHle, Tennessee 

bend IX Carporation 
Southfield, Michigan 

Bethel College 

St . Paul t hinnesots 

BTigha Young University 
Provo, Utah 

Capital Univeraity 

Colunbus, Ohio 

Capital Univeraity 

Univeraity Without Walla Frograsi 

Colunbua, Ohio 

Carlow College 
Pittsburgh, ^ennsylv^nis 

Catholic Univeraity of Series 
School of Education 
Hyattaville, Maryland 

Cedar Creat College 
AUentown, Pennaylvania 

The Center for New Televiaion 
Chicago, Illinois ^ 

Chspssn College 
Orsnge, Californis 

Clayton Junior College 
Horrov, Georgia 

College IV, Urand Valley 

State Collagea 
Allendale, Michigan 



College of Ukc CouDty 
Grayalake, lUiooia 

College of Nev Rocbelle 
24ew Bochelle, New York 

College of St. Benedict (3) 
St. Joaeph, Minneaota 

College of Sc. Mary (2) 
Oaaha, NabrtakA 

College of St. Scholaatica 
Duluth, Minneaota 

College of St. Ihcreaa 
Wiaooa» Minneaota 

The Cooper Joion, Cooper Square (2) 
New York, Nev York 

Creighton Uciivcraity (2) 
Oaaha, Nebraakji 

C. W. h)at College (2) 
Greenvale , New York 

Cuyahoga Comnity College (2) 
Pama, Ohio 

Delgado College 

Nev Orl^ana, Louiaiana 

Denison University 
Grsnville« Cbio 

IjtPsul Ubivarsity 
Chicago, Illinoia 

Dminican High School 
Gtaaha, Nebraak« 

Donnelly College 
Kanaaa City, Kaiisss 

Eaatem Connecticut State Univeraity 
WiUiaaiitic , Connecticut 

EHzabethtowr College (2) 
Elf'sabethtown , Pennsylvanis 

El Fsso CosMiaiity College 
SI Peso, Texas 

Eapire State College 
Old Mescbury, Nev York 

Findlsy Collage 
Findlsy, Ohio 

Flawing Rain^ov Univeraity 
Tahlequah, Oklahoaa 

Florida A 4 H Univeraity 
Tallahaaaee, Florida 

Florida State Univeraity (2) 
Tallahaaaee, Florida 

The Frank Uoyd Wright Fotmdation 
Frank Lloyd Wright School of Archiiacti 
Scottadale, Ariaona 

Franklin Univeraity 
Colwbua, Obio 



Governsra State College (2) 
Park Foreat South, lllinoia 

harris-Stove College (4) 
St. Louia, Mi#aouri 

tioly Naaie College 
Oakland, California 

l^uaatonic Coomuaity College 
Bridgeport, Connecticut 

Hudaon Valley CoMinity College 
Troy, Nev York 

lUinoia Inatitute of Technology 
Chic tgo, lllinoia 

IP inoia itate Univeraity (2) 
Norval , 111 inoia 

Indiana Inatitute of Technology 
Fort Wayne, Indiana 

lows Lakea Cowiunity College 
Estherville, Iowa 

John Jteown Univeraity 
Siloas Springe, Ar kanaaa 

k>liet Junior College 
Joliet, lllinoia 

Kamehaaena Schoola ^2) 
Honolulu, Hawaii 

Kapiolani Onunity College (2) 
Honolulu, Hawaii 

Kentucky State Univeraity 
Frankfurt, Kentucky 

Kinc'a College 
Wiikea-Barre , Fennaylvania 

Kirkhof College (2) 
Grand Valley State Collcgea.^ 
Alle-adale, Michigan 

Kirkwood Conuiity College 
Ceda^ Kapida, Iowa 

ueJl^y College (2) 
CaabT'dge, Maaaachuaetta 

Loyola Ubivef^ty 
Chicago, Illintfia 

Mara Hill Collage 

Mara Hill, North Carolin/t 

Mary College (7) 
Biaaiark, North Dakota 

Marywood College 
Scranton, Pennaylvania 

NcKendree College 
Lebanon, lllinoia 

Medgar Eve/a College 
Brooklyn, New York 

Ne^phia State Univeraity 
MMphia, Tenneaaee 
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Mercer Univertity in Atlanta 
Atlanta, Georgia 

hetropoUtan LoIIege (3) 
Hinneapolia, Minneaota 

hetropoUtan State College ^4) 
St . Paul , hinaeaota 

hiaml-Dade Coou&unity College (2) 
hiami, Florida 

Michigan State Univeraity (2) 
Juatm horrill College 
£aat L4inaing, Michigan 

Mid-Plaina Conunity College 
North Platte, Nebraska 

hmneapolia U>«aiunity College 
Minneapoliai Minneaota 

Mount Marty College 
Yankton, South Dakota 

NaperviUe ComaiAity School Cdatrict #203 
NaperviUe, Illinoia 

Naperville North High School 
Naperville, Illmoia 

Nazareth College of Rocheater 
New York, New York 

New York Society for Ethical 

Culture Schoola (2) 
New York, New York 

Itorth AdMa Stata College (2) 
North Adans, Haaaachuaetta 

Northeastern Illinoia University 
Chicago, Illinoia 

Northeastern University (4) 
fiostont Masaachusetta 

Northwest Alabaaa State Junior College 
Phil Unpbell, Alabama 

Northwest Regional Educational Laboratory 
Portland, Oregon 

Northwestern Lniveraity 
Evanstont Illinoia 

Nova College 

Fort Lauderdale, Florida 

Ofiice oi Catholic Education 
Chicago, Illinoia , 

Otterbein College 
Westerville, Ohio 

Our Lady ot Angela College 
Aston, Pennsylvania 

Our Lady of the Lake Univcraity 
San Antonio, Texaa 

Park College 
Parkville, Miaaouri 



Pennaylvania St#te Uaiveraity 
Kniveraity Pane, Pennaylvania 

KhoJe laland ColUge (2) 
Providence, ttiode laland 

Bock Valley College 
Bockford, lUinoia 

St. J9hfi'a UnivAraity 
Colleg«vilie, hinneaot* 

St . Uo ColUge 
St. l«eo, Florida 

St. Louia Iniveraity School of 

Buaineaa (2) 
St . Louia, Miaaouri 

St. Mary of Celle Pariah 
Btrwyn, Illinoia 

St. Mary* a College of Maryland 
St. Mary'a City, Maryland 

St. Xaviar College 
Chicago, Illinoia 

Spartanburg Technical College 
Spartanburg, South Carolina 

Trenton State College 
Trenton, Mav Jeraey 

Tri County Technical College 
Pendleton, South Carolina 

Irinity Chriatian College 
Paloa heHghts, Illinoia 

Trinity College 
Waahington, D.C. 

Univeraity of Evanaville (2) 
Evanaville, Indiana 

Univeraity of Minneaota 
School of Deniatry (2) 
Minneapolia, Minneaota 

Univaraity of Oklahoiaa 
Nonsan, Oklahona 

Urainua College 
Colleg«vill«« Pannaylvania 

Voorheea College (3) 
Denaark, South Carolina 

taaldorf College 
Foreat City, Iowa 

Italter Sundling Junior High 

School 
Palatine, Illinoia 

Waahington International College 
Waahington, D.C* 

Waat Oahu Collega (2) 
Aihaa, Havaii 

til ar ton Cointy Junior ColUge 
Wharton « taxaa 



Williaa H«iney Marper ColUge (4) 
Palatine, Illinoia 

Wright Inatitute (2) 
Berkaley, California 
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Internat ional 



Brock University 
St . Catharines 
Ontario, Canada 

Georgian College ot. Appl ied Arts and Technology 
Orillia, Ontario, Canada 

Inter-American University of Puerto Rico 
San Juan, Puerto Rico 

Pontiticia Universidad Javeriana 
Faculty Interdisciplinary Studies 
Bogota, Columbia, South America 

Sheridan College of Applied Arts and Technology (2) 
Cakville, Ontario, Canada 

Southwest London College 

Center for Higher Business Studies 
London, £ngland 

University of Puerto Rico 
San Juan, Puerto Rico 

University of t^erto Rico at Jdo Piedras 
Guynabo, Puerto Rico 
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Dissemination Through Mailed Materials 



Persons irom a range of institutions have written to us for 
further information, and we have responded by sending materials 
that related to their efforts. 



In September, 1980, 4500 copies of Valuing at Alverno : The 
Valuing Process in Liberal Education (Eariey, Mentkowski & 
Schafet, 1980), which contains extensive references to the NIE 
fuiided r-searcn to validate Alverno' s curriculum, were mailed to 
values educators and to academic deans in higher education across 
the country. Ibe publication of Analysis and Communication at 
Alverno : An Approach to Critica l Thinking (Loacker, Cromwell, 
Tey I Rutherford, 1984), disseminated in 1984, contains 
references to the research findings. 



The following 207 institutions and representative departments 
received materials. 

Institutions Receiving Requested 
Office of Research and Evaluation Materials 



Wisconsin 

Cardinal Stritch Collie 
Milwaukee, Wisconsin 

Department of Public Instruction 
Madison, Wisconsin 

Marquette University 
Department of Psychology 
Milwaukee, Wisconsin 

O 

Milwaukee School of Engineering 
Milwaukee, Wisconsin 



University of Wisconsin - Madison 
Madison, Wisconsin 

University of Wisconsin - Oshkosh 
Oshkosh, Wisconsin 

University of Wisconsin - Superior 
Superior , Wisconsin 

Viterbo College 
LaCrosse, Wisconsin 



Newspapers Inc. 

The Milwaukee Journal 

Milwaukee, Wisconsin 

St . Luke* 8 Hospital 
Milwaukee, Wisconsin 

St. Michael hospital 
Milwaukee, Wisconsin 
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St. Norbert College 
DePere, Wisconsin 
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Abt Aaaociitei, Inc. 
CMbndge, Hai Hichutettt 

Adelphi University 

School ot NurtiQg r 

Urden City, New York 

Alttka Ftcific Univertity 
Continuing Educttion Departaent 
Anchorage, Alaska 

Albertus Hagnui College 
New haven, Connecticut 

teericen Anthropological Aaaociation 
Washington, D.C. 

American College Testing (ACT) 
Iowa City, Iowa 

The Anericsn he^iiatry ot Badiologicsl 

Technologists 
Minneapolis, Hinnesots 

Anheuser-Busch, Inc. 
Corporate Brewing Operations 
St. Louis, Missouri 

Arizona Oepartsent of Education 
B\oenix , Ar icons 

Ar*2ona state University 
Instructional Design 
University Medis Systeas 
Tempe, Arizona 

Association ot Catholic College and 

Universities 
Washington, D.C. 

August ans Co 1 lege/ Sioux Falls College 
Stoux Falls* South Dakota 

fiall State University 
Career Intormation Services 
Hjncie, Indiana 

Ball State University 
Teachers College 
Burns Laboratory School 
Nuncie, Indiana 

Boston College 

Chestnut Hill, Hassschusett s 

BoViing Green State University 
Bowling Green, Ohio 

BrighM Young University 
Provo, Utah 

Bronx Cosasunity College 
City University ot New York 
Bronx, hew York 

Bryn Mawr College 

Bryn Mavr, Peunsylvania 

CCC Mastery Learning Project 
Chicago, Illinois 

Calvin College 

Grand Rapids, Michigan 



Carser Infomstion S/stsa 
Eugane, Oregon ^ 

Center for Applications ot 
tevelopnentsl Instruction 
College Park, Maryland 

Centrsl Michigan University 
>touQt Pleasant, Michigan 

Chapian College 
Orhnge, Calitornia 

Chicago Public Schools 
Departsent of Curriculia 
Chicsgo, Illinois 

The Chronicle of Hiyher Education 
Washington, D.C. 

Clareaouat Craduats School 
ClarcBOUBtt California 

Clsyton Junior College 
Ikmanities Division 
Morrow, Georgia 

The College of Idaho 
Caldwell, Idaho 

College ot St. Benedict 
St. Joseph, Minnssota 

College -of St. Benedict 
Nursing Department 
St. Joseph, Minnesota 

College of St. Scholastica 
Duluth, Minnesota 

College of St. Theresa 
Winons, Minnesota 

Concert Management 
Falls Church, Virginia 

Coppin State College 
Baltimore, Maryland 

Cornell University 
Field Study Office 
New York, New York 

Cornell University 

tkmsn Development and Family Studies 
Ithsca, New York 

Col'abis university 
Teachers College 
Mew York, New York 

Davidson Collsge 
Davidson, North Carolina 

Delsware County ComauDity College 
Media, fennsylvania 

Dapartmaiit of Professioaal legal st ion 
Talahsssee, Florida 

DaPaul University 
School for New Learning 
Chicago, Illinois 



Esstern Oregon Stste Collsgs 
La Grande, Oregon 

The fidu^Csring Foundation 
Besources tor Muman Development 
Ardmore, Fennsylvanis 

The LChical Culture Sahools 
New York, New York 

Fairhaven College 
Bellinghaa, Washington 

Foreet Service 
Washington, D.C. 

Franklin Univereity 
Columbue, Ghio 

Fund for the Improvement 
ot Post-Sscondary Education 
Washington, D.C. 

Callaudet CoII««%> 
Departmsnt of Education 
Washington, D.C. 

George Meany Center tor Labor 

Studies, Inc. 
Tripartite Progrmo for Apprenticeehip and 

Associate Degree in Libor Studiee 
Silver Spring, Maryland 

George Waehington Univereity 
Waehington, D.C. 

Stete of Georgia 

Departmsnt of Offender Sahabilitation 
Assessment and banagmaent Development 
Atlants, Georgia 

Covernore State Univereity * 
Divieion of Communication and Human 

Services 
Park Forsst South, Illinois 

Grand View College 
Dae Moinee, Iowa 

V 

hamline University 
St. Paul, Minnesota 

Harvard Univsrsity 
Cambridge, Maeeachueette 

Harvard University 
Bureau of Study Counsel 
Cambridge, Maeeachueette 

Harvard Univereity 

Center for Moral Education 

Cambridge, Maeeachueette- 

Harvard Univereity 

harvard School of Dental Medic ins 

Boston, Massachusette 

Haver ford College 
Haver ford, Penneylvania 

Higher Education Daily 
Waehington, D.C. 
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hunter CoUege 

Uiviiion of Progr«mi in Education 
l^ogr«a for Oitted Youth 
New York, New York 

ILL Li ns State University 

DepartiDent ot turriculun «nd Instruction 

Nomil , 11 linois ^ - 

Indiana University 
Sciiool ot Nursing 
Indisnspolis, Indians 

me Institute for Coianunity Service 
Ktblic Service Fellows Progr« 
treeport, Hsine 

Institute for Studies m Educstionsl 
Hathemst ics 
St. Pcul , hionesots 

International business h&chines 
Corporation (IBM) 
Armonk, New York 

International Public itolicy Hesearch 
Lor porat ion 
licLe^n, Virginia 

low* Regents Universities 
Inter inst Ltut lonal Programs 
Iowa City, Iowa 

Long Island Uiiversity 
A & H Schwartz College of fhamacy 
and health Sciences 
brcoklyn , New York 

Long Island University 
C. W. Post Canter 
GreenV^le, hew York 

Loretto heights College 
University Without Walls 
Denver, Colorado 

Loyola University 
Chicago, Illinois 

>':Ber'and Company 
Boston, hassachusetta 

>k::Kay-Uee hospital Center 

Center for Counseling and Therapeutic 

Serv ices 
Ogden, Utah 

teffiphis State Uiiversity 
College ot bducation 
henphis, lennesaee 

hiami University 
Psychology Oupartaent 
Oxford, Ohio 

Micnigan State University 
Institute for Research on Teaching 
tast Lansing, Kichigan 

MilLi College 
Oakland , Call forma 

honiana State University 
'tozenan, hontsna 

hoorhead State 
Psychology Ltppartvent 
Huorhead , Hinnesuta 
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toraiae Valley CoMunity Collags 
Office ot Institutional tesaarch 
Pslos Hills, Illinois 

Mount Aloys lus Junior Collsge 
Cresson, Pennsylvania 

Hutual of Ckiaha Insurance Coapany 
Gtoahs, Nebraska 

National Evaluation Systeas, Inc. 
JIaherst, Massachusetts 

The National Judicial College 
University of Nevada 
Reno, Nevada 

New York State Education Lepart«ent 
Albany. New York 

The New York Iiaes 
New York, Nsw York 

Newaweak hagatinc 

New York, Vmv York 

The Itorth Carolina School ot Sciancs 

and hatheaatict 
DurhaB, North Carolins 

Northern Illinois University 
Departaent of PhiJoiophy 
DeKalb, lUinois 

Northern Illinois University 
School of Allied Health Professions 
DeKalb, Illinois 

Northern Virginia ConaiAity College 
Extended Learning Institute 
Annandale, Virginia 

Northwestern Univeraity 
£vanstonf Illinois 

Northwestern University 
Center for the Teaching Protassions 
Evans toil, Illinois 

Ohio State University 
Departaent of Psychology 
Colvaibus, OhLo 

Ohio State Uiiversity 
National Center for Research in 

Vocational Education 
Coliabus, Ohio 

Oklahoaa State 'diversity 
Stillwater, C4clshoaa 

Oregon Institute of Technology 
Klaaath Falla, Oregon 

Qrgsnisational Assessaent and 

iKvelopaent 
McLean, Virginia 

Organisational Systea), Inc. 
San Liego, California 

Peabody College of Vanderbilt Universi 
Nashville, Tennessee 

Prints George's Coaaunity College 
Largo, Harylacd 



a 
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tegsnts fcytcrnal tograss 
Oultursl Mucation Center 
Albany, New York 

kichland College 
Student Services 
Dallas, Texas 

koberts .^sleysn College 

Rochester. New York ^ 

Rochester -natitute ot Technology 
Uepartnent of Educational Research 
and Development 
Rochester, New York 

Rochester Institute of Technology 
National Technical Institute tor the Deat 
Rochester, New York 

St. John's Uiiversity 
CoUegevilla, Minnesota 

St. Louis University hedical Center 
Departiient of health Mucation 
St. Louis, Missouri 

St. Peter's College 
Jersey City, New Jersey 

St. Olaf College 

Office of Educational Hesearch 

Northfield, Minnesota 

San Jose State University 
Coaaunity Outreach Options for RN's 
San Jose, California 

Search Institute 
Hinncspolis, Minnesota 

Seton hall Univeraity 
South Orange, California 

Sonoas State Uiiversity 
kohnert Park, California 

Southern Illinois University at Carbondale 
Carbondale, Illinois 

Stephens College 
Placenent Office 
ColiMbia, Missouri 

Stockton State College 
Poaona, New Jersey 

Swarthmore College 
Swarthaore, Pennsylvania 

Syracuse University 
College of Education 
Syracuse, New York 

if/racuse University 

Division of Educational Foundations 

School ot Education 

Syracuas, New York 



leachiDf Sefcarch 

A Divifion of tha Oregon State 

Syatam ol Highar Education 
Homouth, Oregon 

Taxes Tech IMiveraity 
School of Nuraiog 
health Sciencea Center 
Lubbock, Tax a a 

Time Magazine 
Waahington, D.C. 

Tri County Technical Collage 
Pendleton, South Carolina 

u S. Military Acadcny 
Inatitutional Heaearch Office 
(Mat Foint, hew York 

Union College 
Departnent of Psychology 
Schenectady, New York 

the Union for Experiaenting Collegaa 

and Uni'eriitLea 
Cincinnati, OhLO 

United Preaa International (UPl) 
New York, New York 

Univeraity of Alab«a^ 

Office ot Educational Developaent 

BirainghM, Alabana 

Univeraity ot California - Davia 
University Medical Center 
Family Nurie Practitioner Prograe 
Sacraaento, California 

University ot California - Loa ^gelea 
Los Angeles, CaLitornia 

University of Cincinnati 
College of Dcaign, Architecture, Art, 
and Planning 
School ot Planning 
Cincinnatip Ohio 

University of Colorado 

College ot Utters » Arts and Sciencea 

Colorado Springe, Colorado 

Univeraity of Georgia 
Departaent of Paychology 
Athens, Georgia 

Univeraity ^f Hartford 
«tfeat hart ford, Connecticut 

Univeraity of Ulmoia at Chicago 
Collage of Education 
Chicago, Illinois 

tjniversity of Iowa 
lotfa City, Iowa 

Univeraity ot kentticky 
Departaent ot Higher Education 
Lexington. Kentucky 

Univeraity of Maryland 
College of Education 
College Park, Maryland 

Univeraity of Maryland 
Counaelmg Center 
College Park, Maryland 



Univeraity of Maryland 
Departaent of Paycholcgy 
ColUga Hrk, Marfliad 

Univeraity of Maryland 
bperiantial Uaraloc Prograa 
Hornbaka Ubrary 
College rark. Maryland 

Univeraity of Maaaachuaetta 
Dapartaent of Education and Psychology 
Aaherat, Ma»»«chuaatka 

Univeraity of Michigan 
Departaent of Psychology 
Ann Arbor, Michigan 

Univeraity of Michigan 
School of nan.iatry 
Ann Arbor, Michigan 

Univa^Lsity of Michigan 
School of Education 
Ann Arbor, Michigan 

Univereity of Mionesote 
Minneepolie, Minneaota 

Univeraity of Minnesots 
School of Dentistry 
Minnespolis, '^innesote 

Universi<:y of Minnef»ote - Norrie 
Horria, Minnesote 

University of Missouri - RoUa 
Counaaling Center 
Bolle, Miaaouri 

Univereity of Notre Daae 
Notre Daae, Indiana 

Univeraity of North Otkota 

Office of Instruct ionsl Developaent 

Greod Forks, North Dekota 

Univeraity of Oklahoaa 
Noman, Oklahoaa 

Univeraity of tha Pacific 
College of Acta and Sciencea 
Stockton, California 

Univeraity of the Pacific 
Departaent of Ihilosopby 
Stockton, California 

The Uiiveraity of Bocheatar 
School of Medicine end Dentiatry 
Diviaion of Medicel Uucation 
Bochaater, Mew York 

University of South Flor&de 

College of Educstion 

Psycho logic si and Social Foundationa 

Dapartaant 
Taapa, Ploride 

Univeraity of Utah 

Graduate School of Education 

Salt Lake City, Utah 

Univeraity of VafMont 
Burlington, Vermont 

University of Itaahington 
Saettle, Ifnahington 



Uhiveraity Syataa of Maw MMpshire - 
Dunlop Center 
Our has, Maw Ha».|»abira 

U.S. Maw and World »»Port. 
laahington, D.C. 

U SA To gay 
Washington, D.C. 

Vanderbilt Univeraity 

Inatitute for Public Policy Studiaa 

MaahviUa, Tanneaaaa 

The l<all Street Journal 
Maw York, Msw York 

Waahington Uhiveraity 

Graduate Institute of Education 

St. louis, Missouri 

Weber SCete College 
OKden, Utah 

Welleeley College 
Departaent of Pvychology 
Welleeley, Maaaechueetta 

Miitaan College 

Walla Walla, Waahington 

Tha Wright Inetitute 
Berkeley, California 
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International 

Australian National University 

Office for Research in Academic Methods 

South Bentley, Western Australia, Australia 

BayamoD Technological University College 
Uepartmcnt of Biology 
Bayamon, hierto Rico 

Ben Gurion University 
School of Medicine 
Beer Sheva, Israel 

Deutsches Institut Fur Fernscudien 

An Uer Universitat Tubingen 
lubingen 1, West Germany 

Erasmus Universiteit Rotterdam 
Rotterdam, Holland, Netherlands 

McGill University 

Centre for Teaching and Learning Services 
htontreal, Quebec, Canada 

he Gill University 
Faculty of bJucation 
Montreal, (^lebec, Canada 

McHaster University 

Department of Chemical Engineering 

Hamilton, Ontario, Canada 

Mount Saint Vincent University 
Halifax, Nova Scotia, Canada 

National University of Lesotho 
Institute of Education 
Lesotho, Africa 

Pontificia Universidad Javeriana 
Faculty Interdisciplinary Studies 
Bogota' , Columbia, South Anerica 

St . Clair College 
Applied Research Centre 
Windsor, Ontario, Canada 

University of British Colunbia 

Faculty of Education 

Vancouver, British Colunbia, Canada 

University of Otta*« ^ 
School ot Mxrsing 
Ottawa, Canada 



University of Puerto Rico 
General Education College 
Physical Science? Department 
Rio Piedras, Puerto Rico 

University of Quebec-Montreal 
Sciences De L*Education 
^ntrcal, Quebec, Unada 

University of Regina 
Regina, Saskatchewan, Canada 

University of the Sacred Heart 
Sacturce, Puerto Rico 

University of Waterloo 
Waterloo, Ontario, Canada 

Western Australian Institute of technology 
South Bentley, Western Australia, Australia 

York University 

Educational Development Office 
Dovasview, Gnt£rio, Canada 
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Materials Dissemmatad at Off-Camput Confarences and Seminars 



This final dissemination list conaists of inatitutions that 
were represented (when registrant lists were available) at the 35 
off-campus meetings, conferences and workshops at which Office of 
kesearch and Evaluation members presented. We recorded 310 
institutions and representative deparunents as having received 
dissemination materials at these conferences. 



State, H: '^mal and Intamationa! Inititutions 
Receiving Offica of Rtiiarch and Evaluation 
Prasantattont and Materials at Off-Campus 
Confarancas and Workshops 

1977 to 1984 



Wisconsin 

Beloit Colli;:ge. 
Beloity Wisconsin 

Edgewood College 
Madison^ Wisconsin 

Johnson Coutrols, Inc. 
Milwaukee , Wisconsin 

S* C. Johnson & Son, Inc. 
Racine 9 Wisconsin 

harquette University 
Milwaukee , Wi scons in 

University of Wisconsin - Eau Claire 
Eau Claire, Wisconsin 

University of Wisconsin - Green Bay 
Green bay, Wisconsin 

University of Wisconsin - Madison 
Kadison, Wisconsin 

Univei'sity of Wisconsin ^ Milwaukee 
Milwaukee , Wisconsin 

University of Wisconsin - Parkside 
Kenosha, Wisconsin 

University of Wisconsin - Stevens Point 
Stevens Point, Wisconsin 

University of Wisconsin - Whitewater ^ 
Whitewater, Wisconsin a 
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AbTAhM UldviD Agriculturtl ColUt,*^ 
TittODt Georgit 

Additon School District 
Additon, lllmoit 

Advanced Rctcarch teaourcca Organisation 
bethecda, Maryland 

AlberCaon'a, Inc. 
^laa, Idaho 

City of Maxandria 
Alaxtndria, Virginia 

Alvin - aunity U>ll«gc 
Alvin, Texaa 

Aaarican Aaaociation for Higher Education 
taathington^ D.t. 

AMcrican Aatocitlion oi Collaget for 

Taachar Education 
Waahington» D.C. 

tearican Aatociation of School AdMinittrttort 
Arlington » Virginia 

itaerLcan Association of IMivaraity Profatsurs 
Waahiogton, D.C. 

itecric&n Caat Iron Pipe Coapaay 
Birainghm, Alabat 

The teerican CoUcga Teating Prograa (ACT) 
love City, love 

iteeiican Coincil on Education 

ACL FelloM in AcfedeMic Adsinittration 

Washington, ii.C. 

Aaerican Coiaicil on Education 
Division of Acad«mic Affairs and 

Insit itutional Relations 
Washington, D.C. 

iteerican Cowcil on Education 
Office ot Woaen in higher Educstion 
Washington, D.C. 

iteericsn Forest Institute 
Washington, D.C. 

iteerican lelephone k Tslegrapti (AUT) 
Basking Ridge^, New Jersey 

iteerican University 
Arlingcon, Virginis 

Anderson College 
Anderson, Indiana 

Anheuser-Busch Companies, Inc. 
Corporate Brewing 
St. Louis, Kissouri 

Anheussr-Busch , Inc. 
Baidvinsville, New York 

Anheuser-Busch, Inc. 
WiUiaasburg, Virginia 

Antioch Volwteer Services 
Ht . Rainier, Maryland 



Aquinas CoUsgs 

Grand Rapids, Michigan 

Arco Oil k Cas 
Dallss, Tszas 

Assessasnt Dssigns, Inc. 
Orlando, Florids 

Aassssaent h Development Associates 
Ukawood, Chio 

Association of AMerican CoUegss 
Washington, D.C. 

Ths Athena Corporation 
Bethasda, Msrvland 

Atlantic Chriatian Collags 
Wilson, North Carolina 

Atlantic RAchlisld Co«pany 
Los Angslss, California 

Auburn Univsrsity at ttontgamsry School 
of Business 
Dapartaenl of Naii«S»nt and Merkating 
ItontgOBary* Alabaaa 

Austin Collage 
Shsnaan, Texas 

AutOBObile Club of hichigaa 
Dearborn, Michigan 

AWC/EDRC 

Maxwell Air Fores Alabaaa 

Baldwin-Wallace CoUegs 
Beraa, Chio 

Ball State Ikiivsrsity 
HunciCt Indiana 

Ball Stats Utaivsrsity 
Miitingar CoUegs of tesiness 
Mincia, Indiana 

Baltiaiora Gas 4 Slectric Coapany 
Baltimore, Harylacd 

Bankers Itust Coapdny 
Mew York, Htw York 

Barry University 
MiSBi, Florida 

Baylor University 
Waco, Texas 

Bell Systea Centsr tor Technical fid 
Lisle, Illinois 

Belleville Area College 
Belleville, Illinois 

Bennstt Gnllege 
Greensboro, North Carolina 

Beraa CoUsge 
Bcraa, Kentucky 

L. M. Berry and Coapany 

Diytun, Ohio 



Blue Cross of Massachusetts, Inc. 
Boston, Hissachusetts 

Bosrd of Bolics Coaaissionsrs 
Kansss City, Missouri 

Boise State University 
Boise, Idaho 

Bowling Green Stete University 
Bowling Green, Ghio 

Be\«.ing Green Ststs University 
College ot Kducstion 
Bowling Grssn, Ghio 

Bowling Grssn Stats University 
EDFI Dipartasnt 
Bowling Gresn, Ghio 

Bradley University 
Fsoris, Illinois 

Brighaa Young Univsrsity 
ProYO, Utah 

Brighaa Yotag University 

Vhysical Education Adviseaent Centsr 

Provo, Utah 

California State University * fSiUerton 
FuUarton, California 

California State University and CoUsges 

Systeas office 
long Beach, California 

Calvin Co) lege 

Grand P^pids, Mlchigsn 

Caaeron Univsrsity 
Lawton, Gklihoaa 

Canisius College 
Buftalo» Mew York 

Capital University Without Wails 
Dayton, Ghio 

Carnegie Corporstion of Mew York 
Maw York, Msw York 

Carrier Corporstion 
Syracuas, Mew York 

Case Hrstsm Reserve University 
^Clevalaod, Ghio 

Case Westsrn Reserve University 
Uss Institute of Technology 
Clevelsnd, Ghio 

Case Wsstsrn Reserve Univsrsity 
School of Organisational Behavior 
Cleveland, Ghio _ - . . 



Case Western Reserve University 
Wsstherhesd School of Manageaent 
Cleveland, Ghio 

CantonsviUe ^ixiity CoUegs 
Allantown, Pisnnsylvsnia 
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Cadtr Crttt ColUgtt 
AllantoMci, ^nntylvanit 

Csntcr for EnvirwOiciittl Iducatxor. •nd 

InstructionAl Scrvictt 
Fore »9«rt, FlorLdt 

C«nt«r for Fiertonnal Beaearch aad tevalopicnt 
OHiwC oi Fvraonnal Haoajiaent 
WAthington, I/.c. 

oentral Hiaaouri State Vnivaraity 
iMrraaburg, Hiaaouri 

Cantral State Univara&ty 
Kdaood, Oklahoaa 

GhapBan CoLLega 
Oraoga, Californta 

Qicltcnba dcbooL Diatrict 
ELktna Park, Nnnaylvania 

Qiicaco Scata Ui&varaity 
th&c«co» ILL&nota 

Chriat&an brothara Collage 
Heaphia, Tanncaaae 

CIBA tharaaceuttcal Coapany 
Sum It, New Jcracy 

ClafUn Lollege 
Orangeburg, South Carolina 

Barry h. Coheo k Aaaociatea 
Largo, Florida 

Ihc College Board - Uaahington Oftica 
liaahLiigtoii, D,C. 

College ot HDunt Samt Vincent 
Rive. dale. Hew York 

Collage oi Hew lochalle 
New Rochelle, Hew York 

College ot St . benedict 
St. Joaaph, Minneaota 

The Co" lege oi St. Catherine 
St. Paul, Hinneaota 

College ot St. Franc ta 
Joi&et, lllinoxa 

ColUge oi St. Scholaatica 
Duluth, Minneaota 

The College ot Wooatar 
Wooater, Ohio 

Colorado School of Minea 
Colden, Colorado 

CoMiaii^y Woaen' a Education Project 



Concordia CQll«ga 
Rivar Fof eat , lllinoia 

Conference Deaign 4 Hanagaent 
baahington, D.C. 

Conatortiutt tor the Mvanceaent ot 

Private Higher Bducatloa 
WaahiQgton, D.C. 



Cooper Ubion 

Maw York, tew Vork 

Coro Voundation 
Maw Voi», Maw York 

Corporation for Public troadcaatiog 
Aonanburg Project 
Itaahiogton, D.C. 

Corporation for FabUc Proadcaatiog 
taablogton, X>.C. 

Cottay Collage 
llavaia, Miaaouri 

Cowicil for the Mvancaaant of 
Ixporiantial Uaroiog 
Col«abia, Harylaad 

om 

Maw York, Ibw York 

Doda Cotaty Public Ichcola ^ 
Miai, Florida 

Dmle Hiai Criainal Juatica Aaaaaaant Cantar 
hialaah. Florida 

Dafanaa lqu«L Opportunity Hanagaaant Inatituta 
Patrick Air For^ laaa, Florida 

DaPaul Ubivaraity 
Chicago, lllinoia 

DaPaul Uftivaraily 
CoUaga ol Arte «ad Sciancaa 
^ Qiicago, lllinoia 

DaPaul Univaraity 

College of Arte and Sciancaa 

Chicago, lllinoia 

D.C, Office of Paraonnal 

taahiogton, D.C. 

Developnant Diaanaiona Internet ionAl 

Pitta6urg)i, Nnnaylvania 

Dtchana Col La,'* a 
Carliala, Fannaylvma 

Draw Uttivaraity 
hadiaon, Maw Jaraay 

Umdalk 'OMtaity Collage 
laltisora, Maryland 

gaat Central College 
Union, Miaaouri 

Beat Tasneaaee Steta Ubivaraity 
Collage of Arte end Sciencee 
Jobnaon City, Tenneaaae 

laatam lllinoia Ubivaraitv 
Cherleeton, lllinoie 



Saetern cky Univereity 

lictaaood, Morucky 

Eaetaro Utahington Ubivaraity 
Cheney, iMabiogton 

Ed*c:^^n State GoMiiaity ColUga 
Pi^oo, Ohio 



laton Aaaocietee 

taaclM Ptloa Var4aa. Ulifornia 

Educational Teating Sonrica 
lorkolay. CaUforoia 

Educational Taatiog Service 
haahiogton, D.C. 

Iducational leatiog Service * 
Princeton p Maw Joraay 

Blgin School Dintf^et 
tlgin, lllinoia 

iKcal, lac. 
aorkarillei^ Arknneee 

Bxcel, lac. 

Cmk Brook, lllinoia 

Fodarel Avietion Adainiatratloo 
taaehingtoo, D.C. 

Floride Internet iouel Uiivereity 
nLmi, rioride 

Florida Power Corporetion 
St. taterburg, Floride 

Floride Scete Uaivereity 
Tellahaeeaa, Florida 

FontboiUM College 

gt . Louie, Nieeouri J*i 

Ford Fo'tndecion 
Mew York, Maw York 

Ford Motor Coapany 
Detroit, Nichigeo 

Frenklin Pierce College 
Bindge, Mtf laaH^ira 

Frenklin Ubivereity 
Coliabua, Gbio 

Front bnga Caaatnity Collega 
Ifeetaineter, Colorado 

Fund for the laprovaont of 

A>et-facondery Bducetion 
Itaehingtonp D.C. 

Ganerel Dynaalce 
Blectric Boet Divieioo 
(Aoton, Connecticut 

Ganarel Motore Corporetion 
Detroit, Michigan 

Ganerel Motore Corporetion 
richer Bgdy Division 
liarren, Kicbigeo 

Ganerel Motore Corporation 
1M Aaeably Divieion 
barren, Mich ig en 

George Maeon Ubivereity 
Fairfax, Virginie 




Georgia tepAr** .«iit ot Of Uodar Ifhab 
Atlanta, Gtorgia 

Georgia Soutbarn Collage 
Stateaboro. Georgie 

sjlerbrt'OK NorCh bigh School 
hortbbrook, lUinoia 

Cordon College 
kenhM , Meetechuaatta 

Covarnora Stata iJniveraity 
Park ^or^^t South, lUinoit 

Grand Viav CoUaga 
Da a h>i.naa. lo'^e 

Guattvua Mo>phua College 
'St. Peter, Ninnaaota 

Helittx CoMunxty Collaga 
lMldon,^^rth Carolina 

harding Univcraity 

College ot Arta and Sciancea 

Searcy, Arkanaaa 

harvard Uiiveraity 

h(«rvard Graduett; School of Education 

Ceabriaga, Measachuaetta 

teidelberg College 

liCtin, Ohio 

The harts Corporetion 
Oklahoaa City, (Alahoaa 

bo! land, Jenninga. k Talbot, Inc. 
Cveneton, lllinoie 

tood College 
Frederick. Herylend 

hope Collega 
Holland, HiChxgan 

Houghton College 
Houghton. Mew York 

City ot Houaton 
hout..on. Texee 

Houeton Baptxat Univeraity 
Houston, Taxaa 

Huaan leaourcea Intarnational 
Sen Diago. Celxtornia 

lllinoia Beiiftdictine Cblle^a 
Lisle, Illinois 

lllvnoia C«ntar tor Educational laprovaaant 
Arlington Heighta. lllinoia 

lllinoia Inetitute ot Technology 
Chicago. Uiinoie 

lUinoie State Univereity 
Nomel , tUinoia 

Indiane Uii/eraity 
hlooBington, Indiana 

Indiana Univeraity 
ftuaineaa Plecanaat Ottica 
Blooaiogton, Indiana 



Indiana Ukiveraity 
School ot luaineaa 
Blo<aington, Indiana 

Indiana Uiivaraity 
tort Uayoe, Indiana 

ladiene ttaiveraity 
Iitdi.M«poli«, Indiana 

Indiana Uiiveraity 
Terra H*ute, ladieoe 

ladiena Itoivaraity of ftonaylvenie 
ladiana, Vantt'^ylvanie 

Indiana Utaiveraity - Furdue ttiivaraity 
rort teyne, ladiene 

bidiana Univeraity - Purdue Uiiveraity 
ladianapolia, Indiana 

lodiane Ikiveraity Southaaat 
Haw Albeny, Indiana 

^ Utamational Buaineaa Mtchinea Corporetion 
Afvook, hsv York 

Intametionel luaiMae Mftckinea Cor^^etion 
Ceitheraburg, Maryleai 

IBM - irPA - PACI 
Araonk, Mew York 

Intarnetionel CoUaga 
Mew York, Maw York 

lom Etate Ubiveraity 
Aaaa, Una 

lows Rata Uiiveraity 
Depertaent of fhyaice 
Aaaa, lows 

Jefteraoo Collegs 
HxUsboTo, Miseouri 

Jeftsraon CoiMiBity College 
Louiavilla, E«ntucky 

Jetterson Coufity Echoola 
Louiaville, Kentucky 

John Carroll Ubiveraity 
Univeravty Haigtata* Ohio 

Johnson Cotnty Coaeunity College 
Ovarlend Park, bnsaa 

Joaaay-Beaa Pukliahii^ Coapeny 
lieahington, D.C. 

Kensea City fo\mt 4 Light 
Kenaea City» Miaaotffi 

bnaaa Hewecn College 
Wichita, lAnaaa 

Kenaaa State Univaraity 
Mai^attan, lAnSaa 

Kenaaa State Ubivareity 
Gollege ot Adainiatration 
Manhatten, l«naea 

bnt State tkiiveraity 
College of Arta sad Scitbcss 
Kent, Ghio 



Ring's Acedeay 

Winthrop Marbor , Illinois 

ICrogsr Coapeny 
Corporsts Offics 
Cincinoeti, Ohio 

Kxogsr, Inc. 
Atlsnte, Georgis 

Ihs Laeming Centtr 

Sen Pr«ncisco, Ce«*fomie 

Uvis k CIsrk Coaawity Collsgs 
Godfrey, Illinois 

Lincoln Lend Goaa«nity College 
Springfield, Illinois 

Lisle District 202 
Lisls* Illinois 

Laaberd Public Schools 
Glen Ellyn, Illinois 

Lopes Aasessaent Eenricss, Inc. 
Port litshington. Haw York 

Loretto tttights College 
Damres, Colorado 

Los Angeles Unified School Oietrict 
Los Angelss, C^litorais 

Loyole Univsrsity 
School of Dtnistry 
Haywood , lllinoia 

Loyole Univeraity 
ScKool of Law 
Chicago, Xllinoia 

Lycoaing College 
liillieaaport, Pennaylvenia 

hadiaon Local Schoola 
tanafiald, Ghio 

hainatrasa Acceaa, In^ . 
Johnatotm, hmaaylvania 

hankato State V^iveraity 
hankatOi hinoaaota 

Harriott Corporation 
Waahington. D.C. 

Mary U^^ea Collega 
Mlat Baint, Miaaiiaippi 

Harycreat Collaga 
Davanport * loMa 

^a Haryleod Inatitute 
ialtiaora, Haryland 

haryaouat Manhattan College 
Miw York, haw York 

P. n, Mclntyre 4 Aaaoeiatae 
Elooafield Hilla. Michigan 

Iliad Johaaon t Coapeny 
tvanavilla, Indiana 

HMphia Stata Univeraity 
Itaaphia , tenneaaae 
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HMphia atata Ubivaraity 

Untar lor th« Study of fttghtr Bducatioo 

Hmphia, 1»noe«««« 

Iteaphia Scat* Uaivaraity 
Collaga of 84ucaCioB 
htaphia, Tttnnaaaaa 

Mercy Collafc of Detroit 
Detroit, Michigan 

haaaiah Collaga 
OraothaM, Fauna ylvania 

Iftitropolitan folica Dapartvent 
iMahington, D.C. 

Metropolitan State Uiivareity 
St . Paul , Mimiaeote 

Metropoiiten Technicel CoMiunity College 
Qiiahe, Mebreeka 

MiaMi Dade CoMuoity College 
MiaMi, Florida 

MiaMi Uiivareity « 

Oxford, Ohio 

MiaMi Univareity - Basil ton 
heailton, Ohio 

Michigan >awicipal League 
Ann Arbor, Michigen 

Michigan State Univereity 
Eaet Laneing, Michigan 

Midlende Technicel College 
ColUBbie, South Carolina 

hiaeieeippi Stete Univereity 
Mieeieeippi Stete, Mieeieeippi 

Montgoaery Comty Government 
kockville. Maryland 

^toraine Velley Coaiunity College 
Peloe hille, llliroie 

Horgen Stete Ubivereity 
ftdltiaore, Meryl end 

Mc Hood Coataiity College 
GreehaM, Oregon 

Mt. Merty College 
Yenkton, South Dakote 

Mt . -te . Mary'e College 
loe Angelee, California 

Ht. Vernon Naxarane College 
Mt . Verron, Ohio 

Mitual oi Ctoaha 
Ctoaha, Mebreeka 

lieeh Technicel College 
Itocky Mount, llor:h Carolin* 

national Aaaocietion of Stete Uaivereitiaa 

and Land Grant Collegee 
Waehington, D.C. 

National Collage of Educetion 
Do war a Grove, Ulinoie 



tetionel Congreee of Mtighborbood tamen 
Brooklyn, Hev York 

National Badomnt for the IkMacitiai 
Waahington, D.C. 

Netiooel Inetitute of Uucetion 
iMehington, D.C. 

Mationel Science fouodetion 
tiaehiogton, D.C. 

Retionvide Ineurence Coapany 
ColMBbua, Ohio 

Nasarath Collaga of locheeter 
locheeter, lav York 

Mew (hurch College 

iryn ithyn, Fenmylvania 

Mew Haapehire College 
Menchef'er, Sew Baapahire 

Maw Hopa Pain Canter 
Alhcibra, California 

New York Power Authority 
Itow York, Mew York 

New York Univereity 
New York, New York 

North Carolina Central Uiivareity 
DurhaM, North Carolina 

North Carolina State UiAvereity 
Kaleigh, North Cerolina 

Ikirth Nannapin CoMuoity Collage 
Minnaapolia, Minnaeote 

North Ideho College 
Coeur d'Alana, Idaho 

North tllinoia Gae 
Aurore, Ulinoie 

North Perk Collage 
Chicago, Ullaoia 

North tixee 8t«Le Uiivareity 
Denton, Xex^e 

Northeeet Mieaouri State Uiivereity 
Xirkaville, Mieeouri 

Northaaetem Ulinoie Uiivereity 
Chicago, Ulinoie 

Ibrthem Ulinoie Uiivereity 
DaKalb, Ulinoie 

Northern Uliaoia Uiivereity 
Collaga of Prof»aaionel Studiee 
DaKalb, Ulinoie 

Northern Kentucky Uiivereitv 
Dapartaant of Educetion 
Bighland Naighta, Kentucky 

Northwaetam Uiivareity 
ivaneton, Ulinoie 

Northwaetem Uiivareity 
School of Rdueetion 
Bv eneton , U 1 inoie 



Nbve College 

Fort Lauderdele, Floride 

Oakland Uiviereity 
locheeter, Michigan 

Oaklend Uiivereity 

College of Arte en4 Sciencee 

locheeter, Michigan 

Office of Menageaant and Budget 
l«aahington, D.C. 

Office oi Pereonnel Hanagaaant 
Uaahington, D.C. 

Ohio ittdicel Indeanity liitual 
liorthiagton, Ohio 

Ohio State Uiivareity 
Coluabua, Ohio 

Ohio %ete Uiivereity Collage 
Coluabua, Ohio 

Ohio state Uiivereity - Manafiald 
Manafield, Ohio 

Oklahoaa Stete Uiivereity 
Stillwater, Okleboae 

Old Ooainican Uiivareity 
Norfolk, Virginie 

O'Laery, Brokaw 4 Aaaocietee 
Clayton, Mieeouri 

Olivet Mesarane College 
Kankakee, Ulinoie 

Olyapia Coaaunity Uiit 
St and ford, lil inoie 

Orgenisational Aaeeec::ent 

Developaent 
McLeaa. Virgia^e 

Owana Corning Fibergleee 
Toledo, Ohio 

Paine ^College 
Auguete, Gecrgie 

Penneylvania Stete Uiivereity 
Divieion of Uidergraduata Studiee 
Altoona, Pinneylvanie 

Penneylvenie Stete Univereity 
Itoivereity Perk, Pinneylvanid 

Philip Itorria D.S.A. 
Kichaond, Virginie 

Phoenix College 
Phoenix, Ariione 

Ihoenix Hitoal Life 
Hartford, Connecticut 

Piedaont Virginie Covunity College 
Oierlotte' /ille, Virginie 

Pine Itanor College 

Cheetttut Hill, Meeeechueette 

POcono Covirooaentel Kducetion Center 
Dingaian Ferry, Fenneylvania 
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?oint Loat Collage 
San Diago, CaUtornia 

Fbrc Authority ot Mew York k Mew J«rte> 
New York. Niw York 

PriDCipit Collect 
Eltah, Illinois 

Public BrordcattiQg Sarvici 
^••hinftOQ, D.C. 

Purdue Ibivtrtity 

liett Leftyccce, Indiene 

Purdue Ubiverticy 
School it Ritnacy 
Vi8tt LeteyeCCc, Indieoe 

Purdue UbivtrtiCy - Ccliaet 
haaeoodi Indiana 

Radford Ubtvertity 
Bedford, Virginit 

laymond ballttr^ Collage 

Cincinneti, Ohio 

Regit College 
Denver, Colorado 

fiend Lake College 
Ina, lllmoie 

Ath iieeerve Officere Training Corpe 
Fort Lewie, Waekington 

Reeerve Officere Treining Corpe 

Fort Honroe, Virginie 

City of KichBOod 
RichBond, Virginie 

The Robert R. Ho ton Heaorial Inetitute 
Oloueeeter, Virginia 

Kock lei end A^eenel 
Rock lei end, Illinoie 

Rockford College 
KocAford, IlUnote 

Rooeevelt Ujivereity 
Chicago, lll\noie 

Boeeaont College 
Roeeaont, Penneylvenie 

Roee Lebe 

ColuBl>ue, Chio 

Seteway Stores, Inc. 
Oaklend, California 

St. Edward* e Univereity 
Auetin, Texee 

St Frencie College 
Loretto, Penneylvenia 

St. John'e Ubivereity 'CollegcviIIe 
Collegeville, Mtnneeote 

St. Lawrence U)iver*ity 
Uepartaent ot Peycoology 
Centon, New York 



SC. L)uie CoMiaiky College 
Sc L>uie, Mieeouri 

Sc. L)uie CoMieiity College ' Meraeec 
RirkWDOd, Mieeouri 

St. Louie Ubivereity 
Sc . L>uie , Mieeouri 

St. Nary*e College 
Winooe, Minneeoce 

St. tet|reburg Police Depertaeat • 
St. Piclceburg, Ploride 

St . niilip*e College 
Sen Antonio, Texee 

SC . Uvier College ^ 
Chicago, Illinoie 

Selea College 

Wioeton-Selea, Morch Caroline 

Sen lemerdino ^iovn-y Schoole 
Julien, Celifcrnie 

Sea Diego Scete Ubivereity 
College of Bueineee Adainietretion 
Sen Diego, Califomie 

Seote Clere Ubivereity 
SeaCe Clere, Celiforttie 

Schenectady Coieity Coaatnity College 
Schenectady, llcw York 

Seere Roebuck & Coapeny 
Chicago, Illinoie 

Seettle Ubivereity 
Scettle, iiaehington 

Service Itercheodiee Coapeny 
heehville, Tenneeeee 

rieoa Beighce College 
Adrian, Michigan 

Sky Chaie 

Dellae-Fort Wortb Airport, Texee 

Social Secwity Adainietretion 
leltiaore, Merylend 

Solotoft & Spivek 
Attorneye at Law 
Greet Neck, New York 

Sootheeet Mieeouri Stet% Itoivereity 
Cepe Cirerdeeu, Mieeouri 

Southeaet Mieeouri S^ete Ubivereity 
Student Develo|Bent Center 
Cepe Cirerdeeu, Mieeouri 

Southern fiell Telephone 
Atlente, Ceorgie 

Southern Illinoie Ubivereity 
Gdwerdeville, Illinoie 

Southern Illinoie Ubivereity 
Cerboodele. Illinoie 

Southern Illinoie Ubivereity 
College of Iducetion 
Cerbottdele, Illinoie 



Southern Illinoie Ubivereity 
College of Liberel Arte 
Cerboodele, Illinoie 

Southern Illinoie Ubivereity 
Bng. Hech. 4 Neteriele Tech. 
Carbondele, lUinoie 

Southern Xllinoie Univereity 
Office of ikedeaic Servicee 
Cerbondele, Illinoie 

Southern Illinoie Ubivereity 
Office of Iteecher Educetion 
Cerbondele, Illinoie 

SouChweet Mieeouri State Ubivereity 
Springfield, Mieeouri 

Southweet Texee Stete Ubivereity 
Sen Mercoe, Texee 

Spalding College 
lAuieviHe, Rentucky 

Speed Scientific School 
lAuieville, lentucky 

Spring Arbor College 
Concord, Michigan 

Spring mil College 
Ik^bile, Alebwe 

Stellinge k Aaeociatee 
Blooaingdele* Illinoie 

The Stenderd Oil Coapeny 
Clevelend, Ohio 

Stenford Ubivereity 
Stenford, Celifornie 

Stete Boerd for Coeiiuiity Collegee 

end Occupetionel Sducetioj^ 
Denver, Coloredo 

State Ubivereity College et buffelo 
Buffelo, New York 

Stete Ubivereity College at Pletteburgh 
Fletteburgh, New York 

Scete Univereity of N^w York 
Cortlend, Mew York 

Stete Ubivereity of Mew York - Oewego 
Oawego, Mew York 

Stockton Stete College 
Poaone, Meu Jereey 

Syrecluee Ubivereity 
Syrecuee, New York 

Teape Electric Coapeny 
Taape, Floride 

Teaple Ubivereity 
Efiiledelphie, Penneylvenia 

Tenneeeee V/illey Authority 
Knoxville, Tenneeeee 

Texee A & H Ubivereity 
College Stetion, Texee 
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Texas Christian University 
Forth taorth, Texas 

lexss Stste Coaptroller 
Trsining Division, DOB 
Austin, Texas 

Thoass A. Ediaon State College 
Trenton, Nev Jeraey 

loMon State Univeraity 
ToMon, Maryland 

Towon State Univeraity 
C/Ollege of Continuing Studiea 
ToKSon, Maryland 

Trenton State College 
Trenton, Nev Jersey 

Trident Technical College 
Qiarleaton, South Carolina 

Trinity College 
Burlington, Ve'^ont 

Triton College 

River Grove, Illinoia 

Troy State Univeraity Syatm 
Florida Region, Inc. 
ttulburt, Florida 

Tulane Univeraity 

New Orleana, Louiaiana 

Tulane Uiiveraity 

College of Arta and Sciencea 

New Orleana, Louisiana 

Union College 
Lincoln, Nebraska 

U.S. Department ot bducation 
Office of the Undersecretary 
Waahington, D.C. 

U. S. Department ot Education 
Poat^Secondary Education 
Waahington, D.C. 

U.S. Department of State 
Waahington, D.C. 

U.S. Foatal Service 
(•aahington, D.C. 

The Univeraity of Akron 
Akron, Ohio 

Univeraity of Alabaia 
Tuacaloosa* Alabama 

Univeraity ot Al«o«a 
Biimingham, Alabama 

univeraity of Alabama 
Department of Biology 
Biimingham^ Alaoaia 

Univaraity of Arixonn 
T^aon, Ariaona 

Univeraity of Ariiona 
College of Bua .eaa 
l\icson. Ariiona 



Univeraity of Arkanaaa 
FayetteviUe, Arkanaaa 

univeraity of Arkanaaa - Littla Bock 
Little Bock, Arkanaaa 

Ubiveraity of Arkanaaa - Pine Bluff 
Pine Bluff, ArkAnaaa 

University of California " Lo« Angel^a 
Loa Angelaa, California 

Univeraity of Central Arkanaaa 
Conway, Arkanaaa 

Univeraity of Chicago 
Chicago, Illinoia 

Univeraity of Connecticut 
Storra, Connecticut 

Univeraity of Delaware 
Newark, Delaware 

Univeraity of £vanaville 
Evanaville, Indiana 

univeraity of Georgia 
Athena, Georgia 

Univeraity of Georgia 
Department of Management 
A£hena, Georgia 

Univeraity of Georgia 
Department of Pa yc ho logy 
Atb«na» Georgia 

Univeraity of Houston " Univeraity Park 
Bouaton, Taxaa' 

Univeraity of lowia 
loua City, Iowa 

Univeraity of Kentucky 
Lexington, Kentucky 

Univaraity of Louiaville 
Louiavillc, Kentucky 

Univeraity of Maine 
Mt . Vernon , Maine 

Univeraity of Maine - Orono 
Orono, Maine 

Univeraity ot Maryland 
College Fftrk, Maryland 

Univeraity of Maryland 
College of Education 
Collaga Park, Maryland 

Univeraity of Maryland 

Inatitute for Experiential Education 

College Park, Maryland 

Uiiveraity of Maaaachuaetta - Boaton 
College of Public and Commtmity Service 
Needham, Maaaachuaetta 

Univaraity of Michigan 
Ann Arbor, Michigan 

Univaraity of Mittttaaota 
Minnaapolia, Minnaaota 



Univeraity of Miaaouri - ColUibi* 
Colunbia, Miaaouri 

Univeraity of Miaaouri - Coluibia 
College of B & PA 
Colwbia, Miaaouri 

univeraity ot Miatouri - Kanaaa City 
Kanaaa City, Miaaouri 

Univeraity of Miaaouri * St. Louia 
St. Lou:,a, Miaaouri 

university of Montsna 
School ot Law 
Missouls, Montsns 

University of Nebrsska - Lincoln 
Lincoln, Nebrsska 

University of New Hampahire 
Durham, New Uamahire 

Univeraity of New Mexico 
Albuquerque, New Mexico 

Univeraity of New Mexico 
College of Engineering 
Albuquerque , New Mtxico 

univeraity of New Mexico 
School of Medicine 
Albuquerque , New Mexico 

Univeraity ot New Mexico 
Loa Alamoa, New Mexico 

Univeraity ot North Dakota 
Grand Forka, North Dakota 

univeraity of Northern Colorado 
Greeley, Colorado 

Univeraity of Northern Io%fa 
Cedar Falla, low* 

Univeraity ot Oklahoma 
Noiman, Oklahoma 

Univeraity of Pittaburgh 
Pittaburgh» Pennsylvania 

univeraity of Puget South 
Tacoma, Waahington 

Univeraity ot San Diego 
College of Arta and Sciencea 
San Diego, Calitornia 

Univeraity of South Dakota 
Springfield, South Dakota 

Univeraity of South Dakota 
Veimillion, South Dakota 

Univaraity of South Alabaaa 
Mobile, Alabaaa 

Tha Univaraity of the State of New York 
Albany, New York 

Univaraity of Tenneaaee 
Knoxville, Tanoaaace 

Univaraity ot Tenneaaee - Martin 
Martin, Tenneaaaa 
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University of Texas 
San Antonio, Texss 

University ot Texas - Arlington 
Arlington Texss 

University ot Texss - Austin 
Austin, Texss 

University ot Tulsa 
Tulsa, Oklahoaa 

University ot lulsa 
College ot Business 
TVilsa, Oklahona 

Univeraity ot Vermont 
Burlingtcn, \erBonn 

University ot Vermont 
College ot Arta and Sciencea 
Burlington, Vcraont 

Univeraity ot Vermont 
State Agricultural College 
Burlington, Vermont 

Univeraity ot Waahington 
Seattle, Waahington 

Univeraity Without Walla ' 
Santa Monica, Calitornia 

Univeraity ot Wyoming 
Laramie, Wyoming 

The Upjohn Company 
KalAaioo, Michigan 

^ Utah Technical College - Salt Lake 
Salt Lake City, Utah 

Valparaiao Univeraity 
Vslparsiso, Indiana 

Vslparsiao University 
College ot Arts and Sciencea 
Valparaiao, Indiana 

Virginia State University 
Peterahurg, Virginia 

Tne Vocation Agency 
New York, Mev York 

Voluie Shoe Corporation 
Topeka, Kansaa 

Walta Walla College 
College Place, Waahington 

Welter Head Army Medical Center 
Department ot Faatoral Care 
Waahington, U.C. 

Waahburn Univeraity ot Topaka 
Topeka, Kanaaa 

the Waahington Center 
Waahington, D.C. 

Waahington Univeraity 
Collage of Arta and Sciencea 
St. Louia, Miaaouri 

Wayne State Collage 
Wayne, Nabraaka 



Webater College 
St. Louia, hiaaouri 

Wellneaa Progm - Educational 
Cooperative Service Unit 
hinneapolia, Minneaota 

West Virginia Univeraity 
Morgantowi, Weat Virginia 

Weat Virginia University 
College of Arta and Science^ 
horgantoun. West Virginia 

Western Carol ma University 
Cullovhee, North Carolina 

Weatern Illinoia Univeraity 
Macomb, iUiaoia 

Weatern Kentucky Ui&veraity 
Department ol Mathematica 

Auburn, Kentueky^^ 

Weatern Kentucky Univeraity 
Bovling (^een, Kentucky 

Weatern Micbigan Univeraity 
Kalamasoo, Michigan 

Weatern Montana College 
Dillon, Montana 

Weatern Mev England College 
Springfield, Maaaachuaetta 

Weatinghouae Electric Corporation 
Baltimore, Maryland 

tiiitman College 

Walls Walla, Waahington 

Wichita 5tat« Univeraity 
Wichita, Kanaaa 

William Jewell College 
Liberty, Miaaouri 

Winthrop College 

College of Arta and Sciencea 

fiockhill, South Carolina 

Women and Found at iooa 
Corporate Ibilanthropy 
New York, hiv York 

Yagar Aaaociatea 
Park City, Utah 

York College 
Jamaica, Mew York 



International 
BHF 

Manpower Planning and Develo|ttent 
Melbourne, Victoria, Auatralia 

Britiah Xelecomnunicationa 
London, England 



Council tor National Academic A/arda 
London, England 

0 I T P 

Tilburg, bollsnd, Netherlands 

John Abbott College 
Bellevue, Quebec, Canada 

Kienbaum Consulting 
Connerbsch, Germany 

Management Service Center Company, Ltd. 
Sniouya, Tokyo, Japan 

^rtbeast London Polytechnic 
School tor Independent Studiea 
London, Ingland 

Northern Telecom 
Brampton, Ontario, Canada 

Public Service Commiaaion 
Covcrnment of Canads 
JOttawa, Ontario, Canada 

Government ot (^ebec 
Defice, Quebec, Canada 

(Xiebec Police Inatitute 
Nicollet, Quebec, Canada 

leader* a Digeat Canada 
Montreal, Canada^ 

Saudi Arabian Airlinea 
Jeddah, Saudia Arabia 

Swediah Management Croup 
iftilmo, Sweden 

Tele~Univeraite 
Quebec, Canada 

lhamea Polytechnic 
London, England 

World College 
London, Englsnd 



Civil Service 
London, England 

Concordia University 
Montreal, (^b«c, Canada 
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ALVERNO PRODUCTIONS PUBLICATIONS 
ORDER FORM 



An overview and ratjopiale for Alverno's approach to the study of college outcomes and a summary of the 
results from the following series of ten research reporis, are found m* Please send me 

rhp following 

Mentkowski, M., & Doherty, A. Careering After College: Establishing the Validity 
Of Abilities Learned m College for Later Careering and Professional Performance. Number 
Final Report to the National Institute of Education: Overview and Summary, 1984 
222 pages '($12.00) 

The research reports described in the overview and summary are: 

One. Friedman, M., Mentkowski, M., Earley, M., Lcacker, G., & D\ei, M. Validating 
Assessment Techniques in an Outcome-Centered Liberal Artb Curriculum: 

Valuing and Communicat»ons Generic Instrument, 1980, 54 pages {$ 4.00) 

Two: Friedman, M., Mentkowski, M., Deutsch, B., Shovar, M.N., & Allen, Z. Validating 
Assessment Techniques in an Outcome-Centered Liberal Arts Curriculum: Social 
Interaction Generic Instrument, 1982, 78 pages ($ 6.00) 

Three: Assessment Comrriittee/Office of Research and Evaluation. Validating Assessment 
Techniques in an Outcome-Centered Liberal Arts Cu^'riculum: Insights From the 
Evaluation and Revision Process, 1980, 5 pages (S 1.00) 

Four: Assessment Committee/Office of Research and Evaluation. Validating Asses'-nent 
Techniques in an Outcome-Centered Liberal Arts Curriculum: Integrated 

Competence Seminar, 1982, 58 pages ($ 4.00) 

Five: Assessment Committee/Office of Research and Evaluation. Validating 

Assessment Techniques in an Outcome-Centered Liberal Arts Curriculum: 

Six Performance Characteristics Rating, 1983, 74 pages ($ 5.00) 

Six: Mentkowski, M., & Strait, M. A Longitudinal Study of Student Change in Cognitive 
Development, Learning Styles, and Generic Abilities in an Outcome-Centered 
Liberal Arts Curriculum, 1983, 357 pages ($18.00) 

Seven: Much, N., & Mentkowski, M. Student Perspectives on Liberal Learning at Alverno 
College. Justifying Learning as Relevant to Performance in Personal and 

Professional Roles, 1982, 83 pages ($ ^.00) 

Eight: Mentkowski, M., Much, N,, & Giencke-Holl, L. Careering After College: 

Perspectives on Lifelong Learning and Career Development, 1983, 124 pages ($ 8.00) 

Nine: Mentkowski, M. DeBack, V., Bishop, J., Allen, Z., & Blanton, B. Developing 

a Professional Competence Model for Nursing Education, 1980, 74 pages ($ 6.00) 

Ten. Mentkowski, M., O'Brien, K., McEachern, W., & Fowler, D. Developing a 
Professional Competence Model for Management Education, 1982, 

317 pages ($20.00) 

Also available from Alverno Productions: 

Mentkowski, M., Moeser, M., & Strait, M. Using the Perry Scheme of Intellectual and 

Ethical Development as a College Outcomes Measure: A Process and Criteria for Judging 

Student Performance. Vols. I & II, 1983, 516 pages ($30.00) 

Mentkowski, M O'Brien, K , Cleve, L., & Wutzdorff, A. Assessing Experiential 

Lea'-ning: The Learnmg^incident as arr Assessment Technique, 1983, 48 pages <$ 4.00) ■.- 
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ALVERNO PRODUCTIOraS PUBLICATIONS, CONTINUED 



Please send me the following: 



• LjbgIQj_Lgy"' "g Alverno College ($ 5.00) 

• Assessnienj a tAjverno Colle_ge ($ 6.00) 

• Nyrsj^ngjducation at Alverno College: A Liberal Arts Model ($ 6.00) 

Valuing atAlverno: The Valuinfl Process in Liberal Education .... ($ 6.00) 

• Valui ng E ducatio^n Materials (packet includes): ($20.00) 

Values Development in Higher Education: A Bioliography 
Generic Criteria for Assessing Levels 1 to 6 of the Valuing 

Competence and Sample Instruments 
Moral Dilemma Materials 

Criteiia for Facilitating a Moral Dilemma Discussion 
Criteria and Worksheets for Designing a Moral Dilemma 
Criteria for Assessing Student Performance in a Moral 

Dilemma Discussion 
Examples of Moral Dilemmas within Specific Disciplines 
A Variety of Modes for Teaching and Assessing Valuing 
The Development of Moral Responsibility in Professional 

Areas of Study (e.g.. Nursing, Business and Management) 

• The Volu nteer As^ssor at jA[verno College ($ 1 -00) 

• Analysis and Communication at Alverno: An Approach to 

Critical Thinking, 19847. ($ 6.00) 



Number 



TOTAL $- 



PREPAYMENT IS REQUIRED. Please mak3 check payable to Alverno College 

Return form to: Alverno Institute, Alverno College, 3401 S. 39th St., Milwaukee, W 53215 



Name: ■ Position: 



Institution/Organization: . Phone ( ). 

Address: 

City State Zip 
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